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EXECUTIVE  SUMMARY 


The  ability  of  para-jumpers(  PJ's)  to  successfully  execute  combat  search  and  rescue  (CSAR)  missions 
depends  on  their  having  persistent  and  comprehensive  situational  awareness.  To  accomplish  this, 
the  PJs  must  effectively  deal  with  many  different  factors.  Particular  challenges  include  navigating 
through  unfamiliar  and  hostile  terrain,  mitigating  and/or  eliminating  threats  to  both  themselves  and 
those  they  are  rescuing,  and  knowing  the  status  and  location  of  their  team  members  and  the  target 
throughout  a  mission.  To  help  meet  these  challenges,  PJs  have  evaluated  and  deployed  different 
technologies,  including  specialized  video  and  heads  up  displays.  The  research  carried  out  through 
this  cooperative  agreement  focused  on  the  continuing  evaluation  of  the  potential  for  synthesized, 
spatialized  audio  to  be  used  as  technology  for  CSAR  missions  capable  of  improving  mission 
performance  by  complementing  existing  audio  communications. 

Humans  can  locate  the  source  of  sounds  very  quickly  and  reliably.  Spatialized  audio  takes  advantage 
of  this  capability  using  software  to  manipulate  an  incoming  audio  signal  so  that  a  subject  perceives 
it  to  be  coming  from  a  particular  location  in  terms  of  its  azimuth  and  elevation.  Because  of  the 
variations  between  individuals  in  terms  of  the  specific  characteristics  of  their  auditory  systems,  such 
as  their  head  size  and  the  shape  of  the  ear,  part  of  the  process  in  exploiting  spatialized  audio  is  the 
measurement  of  the  effect  of  these  factors  in  a  standardized  environment,  such  as  the  Auditory 
Localization  Facility  (ALF)  at  Wright  Patterson  Aid  Force  Base  and  compiling  them  into  a  unique  set 
of  signal  transformations.  This  process  is  well  understood  and  has  already  been  demonstrated  in 
many  experiments.  In  2005,  research  was  started  to  apply  this  technology  to  assist  PJ's  in  CSAR 
missions,  generating  spatialized  signals  to  assist  them  in  navigating  to  a  target  and  being  made 
aware  of  threats  to  them  en  route. 

It  does  not  matter  where  the  actual  sound  source  originates  in  terms  of  how  it  is  perceived  after  the 
software  manipulations.  Knowing  the  position  of  an  individual's  head,  it  is  possible  to  create  a  sound 
that  will  be  localized  correctly.  This  has  created  at  least  two  different  ways  of  using  specialized 
audio.  The  first  is  to  have  the  sound  localized  to  the  actual  speaker,  for  example  a  team  member 
walking  down  another  street.  Another  is  to  have  the  speaker  localize  their  voice  so  it  appears  to  be 
coming  from  a  particular  location  in  the  environment.  For  example,  the  speaker  could  be  an 
observer  at  another  location  on  the  ground,  flying  overhead  or  in  a  command  center  or  interpreting 
sensor  information  from  an  RPV  or  satellite.  Another  possibility  is  that  a  single  audio  signal  could  be 
manipulated  so  that  multiple  individuals  in  a  situation  proceeded  to  come  from  a  common  point  in 
the  environment.  Such  a  capability  would  greatly  simplify  team  communications  and  make  them  far 
more  reliable.  However,  there  is  a  significant  amount  of  additional  research  and  development  that 
needs  to  be  completed  before  an  operational  system  can  be  deployed.  The  research  performed 
during  this  project  is  another  step  in  that  direction,  building  on  a  number  of  previous  research 
initiatives. 

The  experiments  described  in  this  report  that  were  performed  to  evaluate  the  effectiveness  of  a 
spatialized  audio  over  other  audio  communications  were  conducted  using  immersive  virtual 
environments,  sometimes  called  virtual  reality.  While  the  use  of  virtual  environments  for  the 
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evaluation  of  technologies,  like  spatialized  audio,  is  technically  challenging,  it  also  has  certain 
significant  advantages,  including: 


1.  VR  creates  a  compelling,  realistic  environment  in  which  to  perform  activities  comparable  to 
those  that  would  occur  during  an  CSAR  mission, 

2.  Responses  of  individuals  performing  tasks  within  the  environment  can  be  very  precisely 
measured  and  that  information  used  to  analyze  performance, 

3.  Nature  of  the  mission  and  the  environment  in  which  it  occurs  can  be  modified  relatively 
easily, 

4.  Multiple  repetitions  of  particular  task  can  be  performed  in  a  reasonable  periods  of  time,  and 

5.  Non-professionals  can  be  trained  to  operate  in  the  environment  relatively  easily  while  still 
being  able  to  effectively  demonstrate  the  effect  of  the  technology  be  studied. 

The  proposed  research  program  was  to  focus  on  four  particular  areas  related  to  the  operation  and 
application  of  spatialized  audio  within  the  context  of  CSAR  missions. 

1.  Addressing  the  practical  issues  with  respect  to  the  need  for  individualized  audio  profiles  to 
deliver  the  level  of  accuracy  in  identifying  the  locations 

2.  Looking  at  the  effect  of  ambient  noise  on  the  accuracy  of  location  determination  by  subjects 

3.  Evaluating  the  effectiveness  of  different  types  of  external  audio  signals,  i.e.  those  that  might 
come  from  an  observer,  on  the  ability  to  locate  and  eliminate  threats  to  a  vehicle  convoy  in  a 
simulated  urban  environment,  and 

4.  Having  a  PJ  and  target  searching  for  each  other  in  an  urban  environment  while  either 
avoiding  or  eliminating  potential  threats.  This  last  task  also  collected  the  audio 
communication  between  subjects  to  determine  if  having  spatialized  audio  versus  normal 
audio  change  the  nature  of  those  communications  and  their  effectiveness. 

The  technical  development  required  to  create  the  environment  in  which  these  tasks  could  be 
performed  was  extensive.  While  the  technology,  both  hardware  and  software  are  getting  easier  to 
work  with,  the  development  of  effective,  reliable  virtual  environments  with  a  full  range  of 
capabilities  with  respect  to  motion  and  interaction  quires  complex  integration  and  the  development 
of  new  application  capabilities  to  support  component  integration.  An  important  focus  of  this 
particular  project  was  to  find  more  clearly  the  architectural  basis  for  a  multisite,  integrated  virtual 
platform  (IVP)  and  to  start  to  complete  the  development  of  the  necessary  components  within  that 
architecture.  A  great  deal  was  accomplished  and  learned  about  what  the  creation  of  an  effective 
virtual  environment  entails,  but  also  the  tremendous  amount  of  information  that  can  be  derived 
from  it.  Of  particular  note,  was  the  understanding  gained  with  respect  to  the  scope  and  complexity 
of  data  that  virtual  environments  can  and  do  generate  in  the  course  of  an  experiment.  Well  not  fully 
solved,  this  particular  challenge  is  now  better  understood. 

The  results  of  the  experiments  demonstrate  that  spatialized  audio  does  require  individualized  audio 
profiles  to  operate  accurately  and  reliably.  In  addition,  the  vehicle  convoy  and  search  tasks 
demonstrated  that  spatialized  audio  delivers  significant  improvements  over  standard  monaural 
communications,  but  there  was  not  time  to  evaluate  what  impact  ambient  noise  might  have  on  the 
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practicality  and  effectiveness  of  the  spatialized  audio  technology.  However,  enough  was  gained  to 
suggest  that  additional  research  into  the  use  of  specialized  audio  would  be  fruitful,  and  that  the  use 
of  virtual  environments  provided  an  effective  vehicle  in  which  to  perform  meaningful  experiments 
on  the  specific  issues  that  could  lead  to  the  eventual  operationalization  of  the  specialized  audio 
technology. 

Another  result  of  the  work  was  the  development  of  a  generalized  model  to  link  mission  related 
performance  with  human  performance  using  virtual  environments  as  one  vehicle  to  do  this.  The  key 
element  is  to  understand  the  drivers  from  a  human  perspective  that  results  in  improvements  in 
performance  against  mission  objectives.  Without  this,  the  process  of  improvement,  particularly 
where  a  human  is  "in  the  loop",  is  become  "hit  or  miss".  This  is  not  applicable  approach,  a  point  that 
is  made  abundantly  clear  in  the  United  States  air  forces  strategic  document  "Technology  Horizons" 
that  establishes  the  need  to  increase  human  performance  as  one  of  the  key  outcomes  from  the 
plan.  The  work  with  respect  to  CSAR  missions  and  the  performance  of  PJs  represents  only  one  small 
opportunity  to  develop  methods  that  follow  this  approach  and  apply  technologies,  such  as  the  use 
of  virtual  environments,  2  very  quickly  discern  potential  opportunities  for  significant  improvements 
and  to  define  the  path  forward  both  for  research  and  the  commercialization  all  of  appropriate 
technologies. 
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1 .0  SECTION  1 :  SUMMARY 

This  final  report  breaks  down  the  work  that  was  done  in  cooperative  agreement  FA8650-10-2-6048 
into  the  following  sections: 

SECTION  2:  BACKGROUND 

This  section  discusses  previous  work  that  was  done  with  respect  to  the  impact  of  spatialized 
audio  on  performance  in  CSAR  missions  using  simulations  in  virtual  environments  that  led  to 
the  current  program. 

SECTION  3:  PROGRAM  MANAGEMENT 

This  section  deals  with  the  activities  and  approaches  used  to  manage: 

•  Program  activities 

•  Financial  aspects  of  the  program 

•  Technical  development  required  to  support  the  research,  and 

SECTION  4:  TECHNICAL  PROGRAM 

This  section  covers  the  technologies  used  in  the  Integrated  Virtual  Platform  (IVP),  its  physical 
and  application  architectures,  and  the  development  and  integration  of  hardware,  and 
software  components  to  support  the  research  task.  The  section  also  includes  a  summary  of 
the  results,  conclusions  and  recommendations  based  on  the  technical  activities  undertaken 
during  the  program. 

SECTION  5:  EXPERIMENTAL  TASK  REPORTS 

This  section  describes  the  three  experimental  tasks  that  were  completed  including  the 
method,  results,  and  discussion.  The  reports  for  Tasks  3  and  4  are  in  the  format  of  the 
technical  papers  that  were  developed  from  the  work  that  was  done  and  submitted  for 
presentation  at  upcoming  conferences. 

SECTION  6:  RESULTS,  CONCLUSIONS,  AND  RECOMMENDATIONS 

This  section  separately  summarizes  the  broad  findings  and  implications  of  our  technical 
development  efforts  and  experimental  research. 

SECTION  7:  APPENDICES 

This  section  includes  requirements  and  other  supporting  technical  documentation. 

SECTION  8:  GLOSSARY  OF  TERMS 
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2.0  SECTION  2:  BACKGROUND 

Problem 

Threats  on  a  battlefield  can  emerge  quickly  and  without  warning.  Soldiers  or  observers  need  to  be 
able  to  coordinate  with  one  another  in  order  to  locate  and  address  threats  as  quickly  as  possible. 
Ground  or  air  assets  must  be  able  to  direct  friendly  forces  to  a  threat  in  order  to  protect  their  allies 
and  effectively  neutralize  the  threat.  In  order  to  accomplish  this,  speakers  must  be  able  to  convey 
important  points  of  mutual  interest  (Ephrem  et  al.,  2008).  Such  coordination  requires  that  the 
speaker  know  the  whereabouts  of  friendly  forces  and  hostile  threats  and  orient  friendly  forces  to 
the  threat  once  it  is  located.  Spatial  audio  has  shown  great  promise  in  both  communication  and 
navigation,  and  we  believe  it  holds  promise  for  target  identification  and  localization,  as  well. 

Display  options 

The  current  method  of  conveying  location  information  is  verbal  description.  These  descriptions  can 
be  effective,  but  there  are  several  drawbacks  that  make  them  less  than  ideal.  Verbal  descriptions 
are  complex  and  time-consuming.  Descriptions  may  be  ambiguous  and  soldiers  in  different  locations 
relative  to  a  threat  or  point  of  interest  require  different  descriptions,  further  complicating 
coordination  (Ephrem  et  al.,  2008).  Threat  information  can  be  time-sensitive,  and  location 
information  during  battle  must  be  real-time,  intuitive,  and  unambiguous  in  order  to  be  effective 
(Ephrem  et  al.,  2008).  Fortunately,  recent  technological  advancements  have  made  it  possible  to 
present  soldiers  with  real-time  information  about  the  environment  (Gilkey,  Simpson,  Brungart, 
Cowgill,  &  Ephrem,  2007).  There  are  several  potential  methods  of  delivering  such  information  to  the 
warfighter. 

Verbal  descriptions.  Global  Positioning  System  (GPS)  coordinates,  visual  displays,  and  spatial  audio 
displays  are  all  viable  options  for  delivering  real-time  information  to  warfighters  on  the  ground. 
Audio  displays  offer  several  advantages  over  other  display  types,  however.  As  previously  discussed, 
verbal  descriptions  are  complex,  time-consuming,  and  potentially  ambiguous.  Different  descriptions 
are  required  for  warfighters  in  different  locations  relative  to  the  point  of  interest  (Ephrem  et  al., 
2008).  GPS  coordinates  are  exact  and  agnostic  as  to  warfighter  location,  but  conveying  GPS 
coordinates  requires  complex  communication  that  may  be  vulnerable  to  transcription  error,  and  still 
requires  receiving  soldiers  to  calculate  their  own  GPS  position  relative  to  the  threat  (Ephrem  et  al., 
2008).  Presenting  information  on  a  map  or  other  visual  display  is  unambiguous,  but  may  be 
vulnerable  to  map-environment  translation  error.  Map  displays  also  prevent  heads-up,  hands-free 
operation,  requiring  the  warfighter  to  divert  attention  away  from  the  surrounding  environment 
(Ephrem  et  al.,  2008). 
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Why  spatial  audio? 


Spatial  audio  displays  offer  several  advantages  over  verbal  or  visual  displays.  Spatial  audio  displays 
take  advantage  of  human  binaural  hearing  to  present  spatial  information  through  auditory  stimuli 
as  it  would  occur  in  the  real  world.  This  allows  the  operator  to  determine  the  location  of  a  sound 
source,  track  multiple  sources,  and  organize  the  environment  more  effectively  than  through  current 
display  systems  (Simpson,  Brungart,  &  Popik,  2007).  While  other  displays  may  be  ambiguous, 
distracting,  or  increase  workload,  spatial  audio  displays  are  intuitive  and  do  not  impose  additional 
processing  demands  on  the  operator  (Simpson,  Brungart,  &  Popik,  2007).  They  may  also  be  less 
distracting  than  other  types  of  displays  (Gilky  et  al.,  2007).  Audio  displays  can  help  reduce  the 
warfighter's  visual  workload  and  can  help  disambiguate  information  (Simpson,  Brungart,  &  Popik, 
2007).  Spatial  audio  displays  can  be  used  for  directional  cuing  of  threats,  target  acquisition,  or 
navigation  (Ephrem  et  al.,  2008).  They  can  also  disambiguate  communication  channels  to  identify 
the  speaker  (Ephrem  et  al.,  2008). 

Auditory  warnings  are  so  attention  demanding  that  changes  in  the  auditory  displays  are  nearly 
impossible  to  ignore.  This  feature  makes  audio  displays  ideal  for  conveying  critical  information 
(Simpson,  Brungart,  &  Popik,  2007).  The  audio  system  can  also  monitor  the  world  in  360  degrees, 
unlike  the  visual  system  (Simpson,  Brungart,  &  Popik,  2007).  This  makes  spatial  audio  particularly 
well  suited  for  cuing  threats  in  a  dynamic,  uncertain  environment,  where  a  point  of  interest  may  not 
be  in  the  direction  in  which  the  listener  is  looking.  Spatial  audio  displays  offer  the  ability  to  project  a 
speaker's  voice  to  the  location  of  a  referent,  allowing  intuitive,  unambiguous,  real-time  coordination 
about  locations.  Spatial  audio  renders  operator  position  irrelevant,  and  allows  heads-up  and  hands¬ 
free  operation  (Ephrem  et  al.,  2008).  Spatial  audio  displays  are  nearly  ideal  for  communicating  the 
location  of  a  threat  to  multiple  operators  unambiguously  and  in  a  short  amount  of  time. 

Spatial  audio  basics 

Spatial  audio  works  with  human  binaural  hearing  to  generate  sounds  that  behave  as  if  they  were 
generated  by  the  surrounding  environment.  Sounds  are  presented  to  the  eardrum  in  the  same 
manner  that  they  would  be  presented  in  the  real  world.  The  brain  interprets  the  artificial  signal  and 
the  natural  signal  in  the  same  way  (Simpson,  Brungart,  &  Popik,  2007).  A  listener's  head,  pinnae, 
and  torso  all  affect  the  way  that  a  sound  presents  to  the  eardrum  and  allow  humans  to  localize 
sound  sources.  Interaural  time  differences  (small  differences  in  onset  time  between  eardrums)  and 
interaural  level  differences  (differences  in  sound  intensity)  also  aid  localization  (Simpson,  Brungart, 

&  Popik,  2007).  The  changes  on  sound  imposed  by  the  pinnae,  head,  torso,  etc.  can  be  captured 
using  microphones  embedded  in  the  ear.  These  measurements  are  used  to  generate  a  head-related- 
transfer  function  (HRTF).  HRTFs  are  used  to  generate  signal  filters  in  order  to  create  stimuli  that 
match  the  real  world  (Simpson,  Brungart,  &  Popik,  2007). 

Spatial  audio  is  implemented  using  GPS  and  a  head  tracker  (Gilky  et  al.,  2007;  Simpson,  Brungart,  & 
Popik,  2007).  Head  trackers  allow  the  display  to  account  for  listeners'  head  movement  in  real-time 
when  generating  signals  (Simpson,  Brungart,  &  Popik,  2007).  Listeners  use  head  movement  to  clarify 
sound  sources  and  to  place  stimuli  in  the  "auditory  fovea"  for  improved  clarity  (Simpson,  Brungart, 
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&  Popik,  2007).  The  ability  to  utilize  head  movement  is  especially  useful  when  making  front-back 
distinctions  (Simpson,  Brungart,  &  Popik,  2007). 

Past  research 

This  study  is  part  of  a  larger  body  of  work  to  assess  the  utility  of  spatial  audio  for  communication, 
navigation,  and  localization  tasks.  Air  Force  Research  Lab  (AFRL)  studies  have  demonstrated  the 
utility  of  spatial  audio  for  disambiguating  multiple  communication  channels,  thereby  reducing 
workload  and  improving  situational  awareness  compared  to  traditional  systems  (Brungart  & 
Simpson,  2002;  Brungart  &  Simpson,  2005). 

Prior  studies  in  the  AFRL  have  compared  standard  audio  to  spatial  audio  using  a  God's-eye  view  of  a 
virtual  maze  with  a  hidden  downed  pilot  (Ephrem  et  al,  2008).  A  ground  operator  searched  for  a 
downed  pilot  in  the  maze  while  avoiding  hostile  forces  and  ignoring  friendly  forces.  Participants 
located  the  pilot  more  quickly  and  were  shot  by  hostile  forces  less  frequently  with  spatial  audio 
compared  to  standard  audio  (Ephrem  et  al.,  2008).  A  study  using  a  similar  scenario  compared 
navigation  based  on  waypoint  navigation  to  navigation  based  on  bearing  information  delivered  via 
visual  display  or  spatial  audio  (Gilky  et  al.,  2007).  Spatial  audio  was  faster  to  completion  than  the 
visual  display,  offering  the  advantages  of  a  decluttered  visual  field  and  increased  situational 
awareness  (Gilky  et  al.,  2007). 

The  current  program  of  study  is  intended  to  determine  the  utility  of  spatial  audio  for  localization 
tasks.  Previous  research  at  the  AFRL  has  demonstrated  that  spatial  audio  can  improve  performance 
during  visual  search  and  localization  tasks.  This  effect  is  especially  evident  when  the  visual  scene  is 
complex,  as  in  an  operational  environment  (Simpson,  Brungart,  &  Popik,  2007). 

The  Integrated  Virtual  Platform  (IVP) 

The  IVP  is  the  environment  used  to  support  the  current  research  program.  This  facility  was 
collaboratively  developed  by  Wright  State  University  (WSU),  the  Wright  State  Applied  Research 
Corporation  (WSARC)  formerly  daytaOhio  and  the  Air  Force  Research  Laboratory  (AFRL)  as  a  flexible, 
extensible  platform  to  conduct  experiments  studying  human  performance  using  immersive  virtual 
environments. 

In  2005,  Wright  State  University  (WSU)  started  the  initial  work  on  using  virtual  environments  for  the 
combat  search  and  rescue  CSAR)  applications.  The  IVP  was  developed  in  2007  in  a  project  funded  by 
Air  Force  Research  Lab  (AFRL)  Sensors  Directorate.  It  linked  together  WSU's  Virtual  Environment 
Research,  Interactive  Technology,  And  Simulation  (  VERITAS)  facility  located  in  building  441  at 
Wright  Patterson  Air  Force  Base  (WPAFB)  with  the  Wright  State  Applied  Research  Corporation's 
(WSARC)  iSpace™  located  in  the  Appenzeller  Visualization  Lab  (AVL)  in  the  Joshi  Center  on  the  WSU 
campus. 

The  platform  used  two  protocols  for  supporting  distributed  simulations,  High  Level  Architecture 
(HLA)  and  Distributed  Interactive  Simulation  (DIS)  combined  with  scenario  and  artificial  intelligence 
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(Al)  elements  using  commercial  off-the-shelf  (COTS)  combined  with  custom  developed  simulation 
and  visualization  applications.  These  elements  were  designed  so  that  they  could  be  run  on  different 
visualization  hardware  with  little  or  no  modification.  The  first  version  of  the  IVP  was  demonstrated 
to  the  program  sponsors  from  AFRL  in  late  2007  and  there  were  a  continuing  series  of  refinements 
to  improve  reliability  and  consistency  of  IVP  operation  while  supporting  experiments  in  2008  and 
2009.  All  these  experiments  were  done  to  assess  the  use  of  specialized  (3D)  audio  displays  in  the 
context  of  CSAR  missions  in  urban  environments. 

In  2009,  WSARC  developed  the  proposal  that  became  the  basis  for  agreement  FA8650-10-2-6048  in 
a  collaborative  development  effort  with  a  team  led  by  Dr.  Robert  Gilkey  from  WSU.  The  title  of  the 
program  was  "  Evaluating  the  Use  of  Auditory  Systems  to  Improve  Performance  in  Combat  Search 
and  Rescue  Missions"  and  the  focus  was  to  continue  the  effort  toward  developing  an  operational, 
spatialized  audio  system  that  could  be  deployed  in  the  field.  During  the  same  period,  WSARC  made 
additional  investments  to  renovate  WSU  VERITAS  facility  with  funds  from  the  state  of  Ohio  3rd 
frontier  Project.  These  upgrades  included  new  servers  and  projector  systems  that  were  compatible 
with  the  ones  used  in  the  AVL. 

Statement  of  Work 

The  Statement  of  Work  (SOW)  for  FA8650-10-2-6048  included  four  specific  research  tasks  that  built 
of  previous  efforts: 

1.  Assessment  of  the  need  for  individualized  audio  profiles  to  be  able  to  use  a  spatialized  audio 
display  reliably  and  accurately. 

This  has  important  operational  considerations  in  terms  of  deploying  such  a  system  in  the 
field.  These  profiles  adjust  the  performance  of  the  spatialization  software  with  respect  to 
Head  Related  Transfer  Functions  (HRTF)  by  taking  into  account  differences  in  the  auditory 
systems  between  individuals  due  to  the  anatomy  of  the  ear,  head  and  other  factors.  If  it 
were  shown  that  a  generic  profile,  or  HRTF  file,  could  be  used,  then  this  would  make 
deployment  far  simpler.  However,  if  individualized  HRTF  files  were  needed,  then  procedures 
would  have  to  be  developed  to  both  create  them  and  make  sure  that  they  were  correctly 
loaded  into  the  software  for  each  user. 

2.  Evaluation  of  the  impact  of  ambient  noise  on  the  accuracy  of  localization  achieved  by  the 
use  of  spatialized  audio. 

This  is  another  important  operational  consideration  since  it  is  clear  that  PJs  and  others  in 
combat  rely  on  their  sense  of  hearing  to  maintain  situational  awareness  and  respond 
effectively  to  potential  threats  in  the  environment. 

3.  Analysis  of  the  effectiveness  of  spatialized  audio  in  team-based  search  activities. 

Comments  from  previous  meetings  with  PJs  indicated  that  understanding  the  location  of 
other  team  members  was  important  and  could  be  very  time-consuming.  Spatialized  audio 
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could  improve  the  situation  by  providing  unique  localized  sounds  for  each  team  member, 
possible  combining  their  voice  with  other  sounds  during  communications. 

4.  Assessment  of  the  impact  of  using  spatialized  command  and  control  communications  on 
mission  performance. 

Because  the  specializing  software  can  manipulate  an  audio  signal  from  any  source  and 
localize  it  correctly  within  a  PJs  environment,  this  could  provide  a  significant  enhancement 
to  communications  being  sent.  This  would  require  a  means  of  annotating  a  location  within 
an  environment  so  that  the  signal  or  voice  would  be  perceived  as  coming  from  that  location. 
This  would  appear  to  be  less  distracting  than  other  alternatives  like  having  to  look  down  at 
the  display  and  so  a  spatialized  audio  display  could  be  an  excellent  vehicle  for  use  in 
situations  with  heavy  task  loads,  such  as  CSAR  missions. 

In  2009,  while  the  WSARC  proposal  was  in  process,  the  WSU  team  conducted  an  experiment  using 
the  IVP  to  evaluate  the  use  of  spatialized  audio  to  support  a  PJ  on  the  ground  from  a  helicopter 
loitering  above  an  urban  environment.  The  results  of  this  experiment  were  inconclusive.  As  a  result, 
it  was  decided  that  the  project  objectives  should  be  modified  to  conduct  more  research  to  verify  the 
performance  of  the  spatialized  audio  display  using  more  sophisticated  virtual  testing  environments 
and  experiments.  Unless  this  was  done,  the  results  of  any  further  experiments  on  the  effectiveness 
of  3D  audio  for  CSAR  missions  would  be  compromised.  This  decision  was  made  in  March  2010  in 
collaboration  with  the  team  and  AF  Program  Manager. 

The  result  was  a  change  in  the  definition  of  the  tasks  outlined  in  the  SOW  for  FA8650-10-2-6048. 

•  Task  1  was  expanded  in  scope  to  include;  Task  1.1  to  validate  and  verify  the  spatialized  audio 
system  used  in  the  IVP,  and  Task  1.2  to  directly  compare  subject  performance  in  the  IVP  with 
that  in  the  Auditory  Localization  Facility  (ALF)  used  by  the  Air  Force  Research  Lab  (AFRL)  at 
Wright  Patterson  Air  Force  Base  (WPAFB).  The  report  on  this  task  is  entitled  "  The  Influence 
of  Listening  Environments  on  Sound  Localization" 

•  Task  2  to  evaluate  with  effect  of  ambient  noise  was  dropped  entirely  due  to  time 
constraints. 

•  Task  4  was  completed  before  Task  3  and  modified  to  be  a  comprehensive  assessment  of  the 
performance  of  spatialized  versus  semantic  (descriptive)  and  monaural  communications. 
Subjects  travelled  in  a  fixed  position  as  part  of  a  convoy  moving  through  an  urban 
environment  and  had  to  deal  with  sniper  attacks.  It  did  simulate  communications  from  an 
external  observer,  such  as  a  command  center  or  UAV  operator,  which  was  part  of  the 
original  Task  4.  Th  report  on  this  task  is  entitled  "  Aurally  Aided  Visual  Threat  Acquisition" 

•  Task  3  was  simplified  to  involve  two  subjects  trying  to  locate  each  within  an  urban 
environment  using  spatialized  and  normal  monaural  communications,  including  the  role  of 
landmarks  in  enabling  the  activity.  The  report  of  this  task  is  entitled  "The  Impact  of 
Spatialized  Audio  on  Team  Navigation" 
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Information  on  these  changes  was  included  in  the  quarterly  reports  that  were  issued  during  the 
execution  of  the  project. 
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3.0 SECTION  3:  MANAGEMENT 


3.1  Program  Management 

Successful  completion  of  the  program  defined  by  agreement  FA8650-10-2-6048  required  close 
collaboration  of  the  teams  from  the  Wright  State  Applied  Research  Corporation  (WSARC),  formerly 
the  Wright  Center  of  Innovation  for  Advanced  data  Management  and  Analysis,  Inc.  (daytaOhio)  and 
the  principal  subcontractor,  Wright  State  University  (WSU).  As  required,  additional  subcontractors 
were  engaged  form  time  to  time  to  address  specific  technical  areas.  Also,  the  WSARC  team  was 
fortunate  to  have  the  sustained  involvement  and  support  of  Dr.  Brian  Simpson,  the  Program 
Manager  and  Mr.  Vic  Finomore,  both  from  Human  Effectiveness  Directorate,  Warfighter  Interface 
Division,  Battlefield  Acoustics  Branch  (RHCB)  of  the  Air  Force  Research  Lab  (AFRL)  at  Wright 
Patterson  Air  Force  Base  (WPAFB)  during  the  entire  program  providing  insight  and  expertise. 

This  program  team  usually  met  every  two  weeks  to  review  the  progress  on  the  definition  and  design 
of  the  experiments  for  each  of  the  tasks  described  within  the  Statement  of  Work  (SOW)  for  the 
agreement  and  the  related  technical  development.  In  addition  to  these,  the  program  team  held 
longer  meetings,  structured  as  workshops  to  complete  more  in-depth  reviews  of  the  experimental 
and  technical  requirements  related  to  a  particular  task.  There  was  a  session  in  late  April  2010  to 
review  the  approach  for  designing  and  developing  a  new  urban  terrain  that  create  larger,  more 
flexible  and  varied  environments.  Another  session  on  August  30,  2010  defined  the  detailed  plan  for 
Task  4.  Throughout  the  course  of  the  work  on  this  program,  members  of  the  team  made  themselves 
available  to  participate  in  these  sessions  and  worked  in  a  collaborative  manner  its  complex 
experimental  and  technical  aspects. 

To  facilitate  the  meetings  and  provide  a  consistent  format  for  the  agenda  and  associated  material, 
WSARC  used  project  funds  to  acquire  a  collaborative  environment  from  the  Mind  Jet  Corporation 
called  Catalyst™.  This  proved  to  be  an  effective  tool  to  support  meetings  and  provide  a  repository 
for  the  different  types  of  documentation  and  materials  that  were  prepared.  The  Catalyst™  site  was 
only  licensed  for  one  year  consistent  with  the  projected  length  of  the  contract  for  all  participants 
and  is  still  maintained  by  WSARC  to  keep  the  material  available.  In  addition  to  the  meeting  notes, 
the  WSARC  program  manager  received  monthly  reports  from  all  subcontractors  and  issued 
quarterly  reports  to  the  Dr.  Simpson,  the  AFRL  Program  Officer,  the  AFRL  Contracts  Office,  and  the 
Administrative  Contract  Officer  responsible  for  oversight  of  this  agreement. 

The  initial  term  of  the  agreement  was  15  months  running  from  November  2009  to  February  2011. 
However,  as  the  result  of  changes  to  the  scope  in  certain  areas  of  the  project,  additional  time  was 
requested  to  complete  the  work  and  issue  the  final  report.  The  WSARC  program  manager  requested 
no-cost  extensions,  initially  extending  the  end  date  of  the  contract  to  June  30,  2011  and  ultimately 
to  January  31,  2012.  Both  these  were  approved  and  the  WSARC  project  manager  is  appreciative  of 
the  consideration  given  to  these  requests  by  the  Program  Manager  and  other  representatives  from 
the  Air  Force. 
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3.2  Financial  management 


Agreement  FA  8650-10-2-6048  was  set  up  as  a  firm  fixed  price  agreement.  WSARC  established  a 
proposed  set  of  internal  rates  for  the  agreement,  which  were  conditionally  approved  and  finalized 
in  2011.  In  addition,  DCAA  completed  an  audit  of  WSARC's  financial  and  time  recording  systems  and 
procedures,  and  deemed  them  satisfactory  in  March  2010.  It  was  agreed  that  WSARC  issue  invoices 
on  a  quarterly  basis  through  Wide  Area  Work  Flow  (WAWF)  in  advance  to  assist  the  company  in 
managing  its  cash  flow  commitments  with  respect  to  the  agreement.  WSARC  adjusted  invoices  in 
the  subsequent  quarter  for  any  discrepancies  in  time  or  billing  amounts.  In  January  2011,  WSARC 
requested  an  adjustment  to  the  budget  reallocating  funds  from  the  WSU  subcontract  and  this  was 
approved.  This  change  did  not  affect  the  overall  value  of  the  contract.  At  the  date  of  this  report, 
final  steps  are  being  taken  to  close  out  the  financial  aspects  of  the  agreement  working  with  the 
Administrative  Contract  Officer  and  his  staff. 

3.3  Technical  management 

To  complete  the  program  and  three  experiments  involved  a  significant  amount  of  technical 
development  and  related  integration  and  testing.  Jeff  Cowgill  from  WSU  was  the  technical  leader  in 
this  effort,  building  on  his  many  years  of  experience  working  in  the  VERITAS  facility  and  also  in 
WSARC's  Appenzeller  Visualization  Lab  (AVL).  WSARC  hired  a  senior  programmer,  Jim  Flooker,  and 
added  a  summer  intern  from  the  University  of  Dayton,  Andy  Giese  who  continued  to  work  on  the 
project  on  a  part-time  basis.  WSU  graduate  students  also  did  excellent  work  throughout  the 
program.  As  needed,  WSARC  also  engaged  subcontractors  from  time  to  time  to  address  specific 
technical  areas.  While  the  technical  team  attended  all  of  the  regular  program  meetings,  they  also 
held  a  separate  set  of  meetings  on  a  regular  basis,  often  on  the  week  between  the  program 
management  meetings,  to  review  progress  on  the  technical  development.  They  also  held  separate 
in-depth  sessions  and  workshops  to  solve  specific  technical  challenges  and/or  establish  detailed 
requirements.  The  team  was  also  responsible  for  all  of  integration  and  testing,  including  the 
detailed  pre-experiment  tests  of  the  virtual  environments.  This  was  a  critical,  but  very  time- 
consuming  activity  on  part  of  the  team  and  additional  comments  on  this  testing  requirement  are 
included  later  in  this  document. 

The  technical  development  work  followed  a  traditional  development  cycle  starting  with  the  design 
activities  based  on  discussions  with  the  research  teams  and  the  development  of  system 
requirements.  The  requirement  documents  are  included  as  appendices  to  this  report.  For  software 
development,  the  team  set  up  a  common  source  code  repository  and  maintained  it,  checking  in  and 
out  particular  elements  as  work  was  being  done  to  complete  them.  Testing  was  a  major  activity  of 
the  technical  team,  and  beyond  the  normal  unit  and  system  level  testing  there  were  the  walk¬ 
throughs  and  reviews  with  the  experimental  teams.  As  noted,  the  team  completely  verified  the 
technical  set  ups  for  all  experimental  trials  to  ensure  that  configuration  variables  and  controls  for 
the  experiment  were  operating  correctly  before  any  subjects  were  tested.  This  extensive  testing 
was  necessary  for  two  reasons: 
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1.  Operation  of  the  IVP  requires  the  coordinated  execution  of  many  different  software  and 
hardware  components  and  associated  data  sources  and  files.  It  was  essential  to  make  sure 
that  these  configurations  were  properly  set  up  and  the  procedures  initiated  in  the  correct 
sequences, 

2.  Subjects  were  not  available  on  call  and  so  it  was  important  to  minimize  the  possibility  for 
technical  problems  when  they  were  in  the  environment  since  making  up  for  such  technical 
problems  was  difficult  and  costly,  and 

3.  Avoiding  technical  problems  limited  repeat  exposures  for  a  subject  to  the  environment  that 
could  result  in  learning  and  affect  results. 

In  the  course  of  the  program,  the  technical  team  also  had  to  validate  and  verify  the  operating 
environment  in  which  the  subjects  performed.  The  need  for  this  became  evident  from  an 
experiment  run  immediately  prior  to  the  start  of  this  agreement.  The  inconclusive  results  pointed  to 
issues  with  respect  to  the  reliability  of  the  basic  hardware  and  software  configurations  that  were 
used  by  the  subjects.  As  a  consequence,  Task  1.1  was  defined  and  completed  to  provide  the  end-to- 
end  validation  and  verification  of  the  audio  environment  and  all  of  the  software  controlling  it.  A 
similar  process  of  validation  and  verification  had  to  be  done  for  the  environment  in  the  AVL. 
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4.0  SECTION  4:  TECHNICAL  PROGRAM 

4.1  Introduction 

The  technical  program  included  the  design,  development,  and  testing  of  the  Integrated  Virtual 
Platform  (IVP)  to  support  the  experiments  to  assess  the  impact  of  audio  display  systems  on 
performance  during  CSAR  missions.  In  turn,  this  involved  the  integration  of  hardware,  software  and 
communications  components,  some  of  which  were  commercial  products  and  others  of  which  had  to 
be  custom  developed.  Another  component  of  the  technical  program  was  providing  immediate 
support  for  the  experiments,  both  the  testing  of  experimental  scenarios  before  any  trials  were 
executed  and  providing  technical  oversight  and  assistance  during  actual  data  collection. 

To  be  effective,  the  IVP  required  an  architecture  to  organize  the  hardware,  software  and 
communications  components  to  deliver  the  range  of  capabilities  required  for  the  proposed  research 
and  experiments.  This  was  developed  early  in  the  program  and  the  team  used  it  consistently  to 
organize  and  manage  development,  testing  and  experimental  activities.  As  a  result,  this  section  uses 
the  developed  architecture  to  organize  and  explain  the  elements  of  the  technical  program. 

The  architecture  is  divided  into  two  main  parts: 

1.  Physical  architecture  -  describes  how  the  various  hardware  and  physical  communications 
components  are  linked  and  this  is  illustrated  in  Figure  1  and  2 

2.  Software  or  Application  architecture  -  describes  how  the  various  programs,  application 
program  interfaces  (API)  and  tools  interact  to  create  and  present  the  virtual 
environments,  then  record  and  analyze  the  experimental  data 

Separate  sub-sections  describe  each  of  these  in  more  details  and  the  last  part  of  this  section  sets 
out: 

1.  Results  of  the  technical  development  activity, 

2.  Conclusions  or  "lessons  learned"  that  can  be  drawn  from  it,  and 

3.  Recommendations  regarding  future  directions  for  the  development  and  use  of  the  IVP 
for  evaluating  technologies  to  improve  human  performance  for  CSAR  and  other 
applications 
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4.2  Physical  Architecture 


The  IVP  consists  of  two  separate  physical  facilities: 

1.  The  VERITAS  facility  in  Building  441  at  WPAFB  which  is  a  5  sided  (4  walls  and  a  floor) 
virtual  environment,  and 

2.  The  AVL  facility  which  is  a  4  sided  (3  walls  and  a  floor)  BARCO  iSpace™  virtual 
environment. 

These  operate  on  independent  LANs  and  are  connected  to  each  other  through  a  WAN  enabled  on 
Internet2.  Routers  at  both  ends  deal  with  protocol  translations  between  the  native  DIS 
environments  in  each  location  and  allow  DIS  messages  to  traverse  the  Internet2. 

Table  1.  Hardware  components 


Component 

VERITAS  facility 

AVL  Facility 

Processing 

5  -  BOXX  Technologies 

4  -  BOXX  Technologies 

Dual  Xenon  E5420 

Dual  Xenon  E5420 

2.5  Ghz 

2.5  Ghz 

Nvidia  Quad  Fx56000 

Nvidia  Quad  Fx56000 

Win  XP  Pro  32  Bit 

Win  XP  Pro  32  Bit 

Other  Servers 

Other  Servers 

1  -  Audio 

1  -  Audio 

1  -  Tracking  Systems  and 

1  -  Tracking  Systems  and 

Devices 

1  -  Simulation  Servers  To 
Support  Al 

Devices 

Visualization 

5  -  Barco  Galaxy  Nw-12 

4  -  Barco  Galaxy  Nh-12 

Projectors 

Projectors 

1200x1200  Pixels 

1400x1050  Pixels 

Real-Id  Crystal  Eyes 

Real-Id  Crystal  Eyes 

Shutter  Glasses 

Shutter  Glasses 

Tracking/Interactives 

Intersense  900  (Acoustical) 

Arrt  Track  (Optical) 

Head  Mount  And  Hand 

Head  Mount  And  Hand 

Communications 

Held  Wand 

Held  Wand 

Data 

WAN  -  Internet  2 

WAN  -  Internet  2 

Audio 

Senheiser  Hmd-280  Xq 

Senheiser  Hmd-280  Xq 

Headsets 

Headsets 

DIS  Messages 

DIS  Gateway 

DIS  Gateway 
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Figures  1  and  2  below  shows  the  hardware  architecture  for  the  IVP. 

VERITAS 


Figure  1.  Hardware  architecture  for  the  IVP 


AVL 


Figure  2.  Hardware  architecture  for  the  IVP 
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The  following  paragraphs  provide  additional  detail  on  the  various  hardware  components. 

Processing 

The  video  processors  in  both  the  VERITAS  and  AVL  facilities  were  upgraded  in  2008  and  2009 
and  are  specifically  designed  to  support  the  high  speed  rendering  of  images  required  to  support 
the  IVP.  WSARC  used  funds  from  the  Ohio  Third  Frontier  program  and  the  Ohio  Department  of 
Development  to  pay  for  these  upgrades. 

Visualization 

The  projectors  in  the  VERITAS  facility  were  upgraded  with  funding  from  WSARC  in  2009  to  DLP™ 
based  units  from  BARCO  compatible  with  the  units  used  in  the  AVL.  Again,  WSARC  used  funds 
from  the  Ohio  Third  Frontier  (OTF)  program  and  the  Ohio  Department  of  Development  (ODoD) 
to  pay  for  this  upgrade  to  strengthen  the  collaboration  with  AFRL,  the  711th  Human 
Performance  Wing  (HPW)  and  the  Human  Effectiveness  Directorate  (RH). 

Tracking 

Tracking  is  an  essential  part  of  the  immersive  virtual  environment,  providing  the  information  on 
both  the  position  (x,  y,  z)  and  attitude  (pitch,  roll,  yaw)  of  individual  and  devices,  i.e.  in  six 
degrees  of  freedom  (6dof)  within  the  virtual  environment.  Tracked  elements  included  the 
subjects'  heads  and  hand  devices  (pointer,  gun,  grabber). 

In  the  VERITAS,  the  head-tracker  was  a  A  6-DOF  tracking  system  (Intersense  IS-900),  consisting 
of  a  head-  tracker  and  a  tracked,  hand-held  Wand  input  device  communicated  with  the  PCs  via 
another  2.4-Ghz  Xeon  PC  (Dell)  over  Ethernet.  A  trigger  button  on  the  bottom  of  the  wand 
allowed  subjects  to  fire  their  weapon. 

Interactives 

These  included: 

•  The  handheld  devices,  a  Flystick  or  Wand  that  subjects  used  to  interact  with  the 
environment  and 

•  The  devices  used  to  control  motion  of  the  subject  within  the  environments. 

Communications 

The  communications  environment  included  two  distinct  types  of  infrastructure  elements: 

1.  Data  network  -  this  supported  the  operation  of  the  IVP  and  allowed  the  recording  of 
interactions  within  the  environment  for  subsequent  analysis.  This  included  local  area 
networks  (LANs)  at  the  two  virtual  sites  within  the  IVP  and  the  wide  area  network  (WAN) 
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network  components  connecting  the  VERITAS  facility  in  Building  441  on  WPAFB  with  the  AVL 
in  the  JRC  on  the  WSU  campus. 


2.  Audio  network  -  this  allowed  voice  and  other  audio  communications  to  and  between 
individuals  to  be  transmitted  over  the  network  and  arrive  in  the  respective  virtual 
environments.  Since  the  goal  of  the  program  is  to  evaluate  the  impact  of  auditory  systems 
on  performance,  the  IVP  has  specialized  audio  equipment  including  headsets  and 
transmitters  that  operate  wirelessly  so  subjects  are  not  constrained  as  they  perform  the 
various  experimental  exercises.  The  upgrading  of  the  environments  to  wireless  equipment 
took  place  in  2007  as  part  of  the  program  funded  by  AFRL  Sensors  Directorate  (RY)  that 
developed  the  IVP. 

One  critical  aspect  of  the  program  was  the  transition  to  a  homogeneous  DIS  environment, 
allowing  the  use  a  technology  called  "DIS  radios"  that 

a.  Improved  the  fidelity  of  the  sound  that  could  be  transmitted  and  received  within  the 
environment  to  support  wideband  audio  signal  needed  to  accurately  elevation 
perception  and  front/back  discrimination  and 

b.  Provided  compatibility  with  other  Air  Force  technologies  for  recording  and  analyzing 
voice  communications. 


3.  DIS  Messages  -  these  messages  traverse  the  WAN  to  synchronize  activities  in  the  respective 
environments.  In  moving  to  an  all  DIS  environment,  it  became  necessary  to  implement  DIS 
gateways  at  each  locations  to  format  DIS  message  into  UTP  for  transmission  across  the  WAN 
and  then  convert  them  back  into  DIS  messages. 
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4.3  Application  Architecture 


The  software  architecture  provides  the  framework  for  understanding  the  software  components  and 
how  they  provided  the  capabilities  needed  in  the  virtual  environment  to  support  the  three 
experiments.  At  a  high  level,  the  architecture  was  developed  to  deliver  the  following  basic  functions 
in  the  IVP. 

1.  Rendering  -  applications  to  create  the  environment  including  terrain,  buildings,  objects,  etc. 

2.  Simulation  -  tools  to  execute  events  and  activities  within  the  environment  involving  subjects 

3.  Motion  -  applications  to  support  motion  of  subjects  through  the  environment  and  the 
monitoring  of  that  including  navigation  and  orientation 

4.  Interaction  -  applications  to  support  interactions  occurring  within  an  environment,  including 
what  areas  were  being  traversed,  contact  with  buildings,  encounters  between  individuals, 
such  as  shooting  events,  and  communications  between  subject  in  an  environment. 

5.  Coordination  -  the  DIS  (Distributed  Interactive  Simulation)  protocol  provided  the  messaging 
backbone  in  the  IVP,  keeping  track  of  entities  and  coordinating  events  within  and  between 
the  IVP  environments,  such  as  starting  and  stopping  events. 

6.  Data  capture  and  analysis  -  a  system  to  collect  and  manage  the  data  created  by  experiments 
in  the  IVP  used  for  performance  analysis  and  event  re-creation. 

The  architecture  of  the  IVP  includes  the  following  elements  as  shown  in  Figure  3  below.  Separate 
sub-sections  describe  the  purpose  of  the  component,  the  key  development  work  that  was  done 
during  this  project,  and  how  it  supported  the  experimental  tasks.  The  task  information  is  presented 

in  the  order  in  which  the  experiments 
were  performed  starting  with  Taskl, 
then  Task  4,  and  Task  3. 

Applications  were  developed  in  C++ 
using  Visual  Studio  2005  (Microsoft), 
Vega  Prime  API  (Presagis),  Dl-Guy 
(Boston  Dynamics),  and  Immersive 
Module  for  Vega  Prime  (Mechdyne). 
Table  1  summarizes  the  overall 
development  activities  within  each 
element  of  the  architecture  and  for 
each  task. 


Figure  3.  Architecture  of  the  IVP 
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Tablel.  Application  Components 


Component 

Common 

Task  1 

Task  3 

Task  4 

IVP 

DIS 

DIS  Routers 

ALF  (Sphere) 

analog 

environment 

Implement  DIS 
Routers 

IEC 

C++ 

IEC  vl.O  with 

ALF  simulation 

controls 

1C  vl.O 

XML  control  file 

IEC  v2.0  with 

convoy/sniper 

simulation 

Scenario  Generator 

XML  control  file 

IVE 

•  Vega  Prime 

ALF  (Sphere) 

Multiple  start 

Convoy  motion  model 

API 

analog 

points  and  Land 

with  pop-up 

•  Immersive 

module  for 

VEGA  Prime 

•  MakVR 

Forces 

environment 

Mark  toggle 

sniper/distractors 

IAE 

SLAB  3D  v6.5.0 

ALF  sound 

DIS  radios 

DIS  radios 

DIS  Radios 

analog 

Multiple  spatial 
audio  displays  for 
localizing  team 
and  annotation 

3  sounds  prompts  - 
mono/semantic/spatial 

IDB 

Mak  Data  Logger 

MAK 

IEC  collection 

MAK 

IEC/IVE  collection 
Audio  file 
collection 

MAK 

IEC/IVE  data  collection 

Terrain 

Presagis 

CREATOR  PRO 

ALF(Sphere) 

analog 

Urban  tiles 

Urban  tiles 

Al 

•  MakVR 

Forces 

•  Boston 
Dynamics  (DI¬ 
GUY) 

Hostiles/civilians 

Sniper/civilian 

distractors 

Treatments 

XML  file 

1  Audio  -  Spatial 

•  Tile 

•  Start  point 

•  Landmark 

•  2  Audiol 

•  Tile 

•  Time  of  day 

•  Path/active  snipers 

•  3  Audio 
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Integrated  Virtual  Platform  (IVP) 


Purpose 

This  is  the  overall  platform  that  allows  the  interoperation  of  activities  between  the  two  virtual 
environments  that  make  up  the  IVP;  VERITAS  and  AVL.  The  purpose  of  this  platform  is  to  facilitate 
the  use  of  virtual  environments  for  a  diverse  set  of  applications,  including  the  testing  of  new  types 
of  technologies  to  improve  human  and  mission  performance.  The  IVP  was  designed  to  provide 
flexibility  in  terms  of  how  environments  are  created,  the  activities  that  can  be  performed  within 
them,  and  the  types  of  technology  platforms  on  which  they  can  run.  Another  important  component 
of  the  IVP  architecture  is  the  inter-site  communications  infrastructure,  including  DIS  routers  at  each 
end  and  the  WAN  and  LAN  connections  to  and  from  these  routers. 

Shared  development 

1.  All  DIS  environment 

The  major  change  in  the  IVP  was  the  implementation  of  a  homogeneous  DIS  environment.  This 
was  done  to  support  the  delivery  of  wideband  audio  signals  between  sites  necessary  to  support 
elevation  perception  and  front/back  discrimination  Previous  projects  used  Team  Speak,  a  COTS 
Voice  Over  Internet  Protocol  (VOIP)  software),  which  has  more  limited  capability.  As  a  result, 
there  was  concern  that  some  of  the  subtler  effects  particularly  elevation  were  being 
compromised.  In  addition  this  change  to  DIS  was  made  to  ensure  compatibility  with  other  AF 
and  RHCB  development  activities,  including  the  handling  of  multichannel  communications. 

To  make  the  transition  to  a  homogeneous  DIS  environment,  the  team  defined  specific  DIS 
messages  to  support  the  design  requirement  of  the  experiments.  In  the  appendices,  there  is 
material  on 

•  The  DIS  protocol,  an  accepted  IEEE  standard,  and 

•  The  various  messaging  elements  used  to  control  and  manage  the  distributed  events  and 
activities  within  the  IVP. 

2.  DIS  gateways 

A  consequence  of  moving  to  the  all  DIS  environment  was  the  need  to  install  DIS  routers  in 
addition  to  the  WAN  routers  used  to  connect  the  VERITAS  and  AVL.  These  routers  or  gateways 
allowed  DIS  messages  to  be  converted  to  User  Datagram  Protocol  (UDP),  transmitted  across  the 
Internet2  using  Transmission  Control  Protocol  (TCP),  and  then  translated  back  into  DIS  messages 
without  any  significant  latency  or  delay  so  that  events  in  the  two  virtual  environments  remained 
correctly  synchronized 

Task  related  development 

Task  3 

This  task  involved  individual  subjects  in  each  location  interacting  within  the  same  virtual 
environment.  Supporting  this  required: 
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•  Implementation  of  the  DIS  routers  to  facilitate  moving  to  an  all  DIS  environment,  and 

•  Completing  the  validation  and  verified  procedures  developed  in  sub  task  1.1  for  the 
audio  environment  in  both  the  iSpace  and  VERITAS  facilities. 

Matt  Middendorf  of  Middendorf  Scientific  Services  Inc.  deserves  particular  thanks  for  the  work 
in  getting  the  DIS  router  software  from  the  Joint  Services  Action  Force  and  helping  the  team  to 
implement  and  test  the  software  to  get  the  connection  between  the  locations  up  and  running  in 
a  timely  manner. 

IVP  Experiment  Controller/IVP  Controller  (IEC/IC) 


Purpose 

During  the  research  conducted  in  2007,  it  became  clear  that  it  would  be  increasingly  important  to 
control  all  of  the  various  configuration  and  experiment  control  elements  through  a  single 
application,  particularly  as  increasingly  sophisticated  experiments  were  planned  for  the  IVP. 
Basically,  it  became  essential  to  ensure  that  when  a  subject  was  in  the  IVP  all  of  the  environmental 
variables  were  correctly  selected  started  and  shut  down  synchronously  and  at  the  correct  time 
during  an  experiment.  While  this  could  be  done  manually,  doing  so  became  less  viable  as  the 
complexity  of  the  environment  increased,  and  the  complexity  and  cost  of  the  experiments  being 
done  increased.  Fully  documented  and  systems  supported  command  and  control  functions  would 
make  the  IVP  more  reliable,  helping  experienced  users  avoid  mistakes,  and  also  making  it  a  more 
attractive  experimental  venue  for  less  technically  skilled  managers  or  supervisors. 

Shared  Development 

The  IEC/IC  was  a  major  development  effort  within  the  program.  Up  to  this  point,  the  IVP  had  a 
limited  controller  to  deal  with  the  following  complex  processes: 

•  Configuring  the  technical  environment,  i.e.  loading  specific  programs  in  a  particular  order 

•  Loading  of  the  correct  experimental  control  files  and  parameters,  i.e.  what  subject,  terrain, 
audio,  Al,  and  treatments 

•  Managing  the  events  to  occur  within  that  particular  experiment  and  trial,  i.e.  starting  and 
stopping  movement,  recording  hits  on  targets,  and  creating  the  related  data  files 

•  Collecting  data  and  messages  during  the  course  of  an  event,  and 

•  Shutting  down  the  IVP  in  an  orderly  fashion 

The  goal  is  to  make  the  IVP  more  reliable  and  error  free  during  experiment  by  providing  a 
comprehensive,  user-friendly,  GUI  based  control  capability.  This  will  encourage  experimenters  to 
use  the  IVP  with  less  training  about  the  IVP.  Specific  development  efforts  included. 

1.  GUI  based  IEC 

As  a  result,  the  creation  of  a  fully  functional  experiment  controller  was  an  early  and  major 
development  goal  of  this  program.  This  goal  was  not  completely  achieved.  The  requirements 
for  an  intuitive,  GUI  based  IEC  are  included  in  the  appendix,  and  a  version  of  this  was  used  to 
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manage  the  execution  of  Task  1  (Sphere  Task)  and  Task  3  (Sniper  Task),  controlling  the 
environments  for  the  experiments  and  initiating  and  controlling  its  execution,  including  the 
collection  of  relevant  data.  Time  and  budgetary  constraints  prevented  the  extension  of  the 
GUI-based  environment  to  support  the  full  (two  location)  IVP  needed  for  Task  3  (Teaming 
Task),  so  a  command  line  controller,  called  the  1C,  was  written,  and  used  successfully.  A  goal 
of  the  next  round  of  development  of  the  IVP  will  be  to  complete  the  GUI-based  application 
to  provide  the  most  reliable  and  error  free  environment  for  all  users,  but  allowing  less  skilled 
or  trained  technical  users  to  set  up  and  run  experiments. 

2.  Extracting  information 

Another  major  development  activity  had  to  do  with  the  use  of  the  IEC  to  execute 
experiments  based  on  the  extraction  of  information  from  the  environment  itself.  For 
example,  the  design  of  Task  4  required  the  identification  of  suitable  locations  in  the  urban 
environments  from  which  to  attack  a  subject  riding  in  a  Humvee.  For  each  subject  and  all 
subjects,  these  threats  had  to  come  from  directions  and  elevations  that  fully  tested  the 
responses  to  different  audio  signals,  so  these  had  to  be  identified  and  selected  in  advance  to 
get  a  correct  distribution  of  locations  and  make  sure  that  the  sniper  would  be  visible.  To 
make  the  task  of  finding  the  sniper  more  challenging,  the  design  called  for  ordinary  civilians 
(distractors)  to  appear  close  to  the  sniper,  so  more  locations  had  to  be  selected  and  tested 
before  the  experiment.  In  turn,  this  information  was  used  by  the  IEC  to  activate  a  sniper  and 
associated  civilians  at  the  correct  point  in  the  Humvee's  movement  through  the  urban 
environment.  Being  able  to  identify  and  reference  specific  locations  is  a  crucial  capability  for 
experiments,  particularly  involving  external  participants,  such  as  an  airborne  observer  or 
UAV  operator. 

3.  XML  definition  file 

To  support  the  operation  of  the  IEC,  the  team  also  developed  an  extensible  XML-based  file 
to  contain  all  of  the  configuration  parameters  for  experiments.  This  provided  an  effective 
vehicle  for  the  control  and  execution  of  sophisticated  experiments.  For  example  in  the 
Sniper  Task,  the  XML  file  contained  information  on  the  subject,  the  audio  files  to  use,  the 
start  and  stop  locations  for  the  trial,  where  and  what  targets  and  distractors  would  be 
presented  to  the  subject,  the  time  of  day  in  the  environment,  along  with  other  parameters. 
The  advantages  of  using  an  XML  file  are  that  it  is  standardized  and  has  an  easy  to  edit 
structure  and  flexibility. 

The  requirements  document  for  the  IEC  is  included  in  the  appendix  to  this  report,  as  are 
samples  of  the  XMLfile. 
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Task  related  development 


Task  1 

This  was  the  first  experiment  in  which  the  IEC  was  used  to  control  the  experiment  and  the 
treatments  for  the  subjects.  Although  simpler  than  subsequent  environments,  it  represented 
a  significant  improvement  over  previous  versions  of  the  IVP  in  terms  of  the  control  it 
provided  for  those  managing  the  experiment  and  ensuring  that  the  correct  parameters  and 
file  sets  were  loaded  prior  to  the  execution  of  the  experiment. 

Task  4 

The  IEC  was  extended  in  Task  4  to  include  the  input  from  the  "scenario  generator"  and  XML 
control  file  to  manage  the  full  set  of  activities  for  the  vehicle  convoy,  including 

•  Moving  from  a  defined  starting  point 

•  Detecting  entrance  into  an  attack  area 

•  Stopping  of  the  vehicle 

•  Presenting  targets  and  distractors 

•  Communicating  with  the  IVE  about  trial  status 

This  was  a  very  significant  level  of  complexity  to  manage  within  the  controller,  but  was 
nonetheless  successfully  implemented.  As  noted,  additional  development  is  needed  to 
improve  it's  ease-of-use  and  reliability  in  terms  of  managing  experiments. 

Task  3 

Time  and  funding  made  it  impossible  to  extend  the  capabilities  of  the  IEC  to  deal  with  two 
IVP  environments  at  the  same  time.  As  result,  a  decision  was  made  to  develop  the  1C,  a 
simpler,  command  line  controller  for  this  experiment  that  could  be  implemented  within  the 
contract  period. 

IVP  Visual  Environment  (IVE) 


Purpose 

The  IVE  is  an  essential  component  for  all  experiments  using  virtual  environments  since  it  is  what 
drives  the  visual  experience  in  real  time  and  in  immersive  stereo.  It  defines  what  subjects' 
experience  in  the  IVP,  including  their  auditory  and  visual  experience  and  interactions  with  the 
virtual  environment.  It  brings  together  the  rendered  elements  of  the  terrain,  what  subjects  see, 
with  tracking  and  interactive  capabilities,  to  allow  subjects  to  move,  see  and  perform  tasks  within 
the  environment  and  other  entities  (avatars)  within  it.  As  a  result  of  the  work  done  in  the  course  of 
this  program,  it  was  also  learned  that  the  IVE  has  to  deal  with  a  portion  of  the  data  collection  tasks 
to  avoid  latency  problems  and  ensure  synchronicity  in  data  between  environments  that  are 
physically  separated.  The  principal  tool  used  to  develop  the  IVE  was  Vega  Prime  from  Presagis. 
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Shared  development 


The  following  were  key  shared  development  efforts: 

1.  Simulation  of  the  ALF 

The  ALF  is  4.3-m  geodesic  sphere  with  277  Bose  11-cm  full-range  speaker  drivers  mounted 
its  surface  (see  Figure  4).  The  sphere  is  housed  in  a  large  double-walled  anechoic  chamber. 
Custom-built  high-power  switching  hardware  (Winntech)  can  route  up  to  15  signals  to  any  or 
all  of  the  277  speakers.  Early  in  the  program,  the  team  decided  to  build  a  replica  of  this  in 
the  VERITAS  environment  to  ensure  that  results  for  experiments  being  done  in  IVP  could  be 
tied  to  outcomes  of  research  that  had  already  been  done  in  the  ALF.  A  virtual  environment 
comparable  to  the  ALF  was  developed  and  the  IEC  programmed  to  simulate  the  visual  search 
and  other  types  of  ALF  activities. 


2.  Expanded  Urban  Terrain 

The  IVE  was  modified  to  accommodate  the  use  of  much  larger  and  more  complicated  urban 
terrains.  In  all  previous  IVP  experiments,  even  one  involving  an  observer  in  a  loitering 
helicopter,  the  scale  and  flexibility  of  the  terrain  was  limited.  In  addition,  the  experiments 
reported  here  the  IVE  had  to  handle  more  sophisticated  interactions  between  the  wand  and 
the  environment  and  support  different  motion  models  in  the  two  environments.  When  the 
problem  related  to  centralized  data  collection  became  evident,  the  IVE  was  enhanced  to 
capture  critical  data  within  each  environment  on  a  local  basis. 

3.  Performance 

Performance  of  the  IVE  is  a  critical  issue  because  the  active  stereoscopic  technology  used  in 
the  IVP  means  that  the  environments  must  be  rendered  at  a  certain  frame  rate  to  create  a 
realistic  environment,  avoiding  distracting  jerkiness  in  the  motion  as  subjects  seek  out 
targets  or  move. 

4.  Motion  models 

Work  during  the  project  also  addressed  the  differences  between  the  two  environments  of 
the  IVP,  i.e.  VERITAS  has  5  sides  and  the  iSpace  has  4  sides.  This  changes  what  subjects  can 
do  in  some  situations,  for  example,  turnaround  to  see  thing,  the  team  created  models  that 
supported  realistic  movement  in  both,  so  that  with  training,  the  environmental  differences 
were  not  an  issue. 
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Task  related  development 


Task  1 

The  major  component  here  was  the  creation  of  the  sphere  environment,  which  replicated 
the  ALF  in  terms  of  its  physical  appearance.  In  addition,  the  IVE  ran  events  that  were 
analogous  to  what  was  normally  done  in  the  ALF. 

Task  4 

The  major  development  was  to  create  a  motion  model  involving  a  convoy  of  vehicles  moving 
along  a  predetermined  path  through  an  urban  terrain.  This  included  a  control  element, 
forcing  the  subject  to  focus  on  a  target  on  the  back  of  the  vehicle  immediately  ahead  of 
them  in  the  convoy,  to  make  the  convoy  move  and  manage  the  orientation  of  a  subject 
when  they  entered  an  ambush  location.  These  locations  were  controlled  to  ensure  that 
snipers  appeared  across  a  full  range  (azimuth  and  elevation)  of  locations  throughout  the 
course  of  the  experiment,  i.e.  a  subject  would  have  to  locate  objects  all  around  them  and 
also  in  elevation  from  where  they  were  to  realize  a  reasonable  distribution  comparable 
across  all  subjects  in  the  experiment.  The  IVE  responded  to  commands  from  the  IEC  to  start 
and  stop  following  the  predefined  paths,  load  a  file  defining  a  path  and  pausing  for  trials. 
Another  major  development  effort  completed  for  this  task  was  the  addition  of  local  data 
collection  in  the  IVE.  This  allowed  for  collection  of  user  inputs  and  the  visual  environmental 
status  at  the  source  without  added  latency. 


Task  3 

This  experiment  was  the  least  constrained  in  that  it  involved  dismounted  subjects  able  to 
move  freely  through  the  urban  environment.  This  is  very  different  than  the  other  tasks  and 
more  representative  of  the  type  of  tasks  that  need  to  be  performed  going  forward  to 
evaluate  potential  technologies,  like  spatialized  audio  displays  that  could  help  PJs  perform 
more  effectively. 

The  IVE  continued  to  use  most  of  the  elements  developed  in  previous  experiments  including 
the  introduction  of  hostile  avatars.  Flowever,  these  now  operated  in  a  far  larger  terrain  than 
was  previously  used.  As  a  result,  shooting  interactions  and  other  things  had  to  be  modified 
because  the  distances  in  the  new  "tiled"  terrain  were  far  greater.  The  walking  motion  model 
for  the  participants  was  updated  to  improve  collision  detection  with  other  objects  in  the 
virtual  world  and  add  support  for  foot  pedals  as  an  input  device  in  the  iSpace. 

Unlike  the  VERITAS  facility  in  which  a  subject  is  completely  enclosed  in  the  simulation,  the 
AVL  has  an  open  back  wall  that  necessitated  a  slightly  different  system  of  moving 
throughout  the  virtual  environment.  Because  the  subjects  could  not  physically  turn  to  face 
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the  rear  of  the  room,  they  used  a  foot  pedal  system  that  allowed  their  avatar  to  turn  in  the 
virtual  environment  while  the  participant  remained  stationary.  This  allowed  them  to  explore 
the  entire  environment  while  maintaining  complete  immersion. 

The  method  for  motion  tracking  differed  by  facility.  In  the  VERITAS,  a  head  tracker  was 
employed,  involving  a  small  bar  on  top  of  the  headphones,  which  was  attached  to  a  battery 
pack  in  the  vest.  At  the  AVL,  however,  head  tracking  was  done  by  (equipment  name) 
cameras  in  the  iSpace,  which  cued  off  of  reflective  balls  attached  around  the  3-D  shutter 
glasses. 

IVP  Audio  Environment  (IAE) 


Purpose 

This  component  of  the  architecture  supports  the  audio  environment  that  subjects  experienced 
during  the  course  of  an  experiment.  This  included  person-to-person  communications  and/or  signals 
received  from  a  command  center  or  server  or  in  the  case  of  Task  1  simulating  various  speaker 
locations  within  the  simulated  ALF  environment  and  its  virtual  replica.  This  is  the  part  of  the 
architecture  that  supports  the  spatialization  of  incoming  audio  signals  so  that  they  are  rendered  to 
come  from  the  correct  location  within  the  virtual  environment.  The  3D  models  and  data  were 
combined  with  simulation  positions  to  update  the  spatialization  in  real  time.  That  is,  sounds  always 
appeared  to  arise  from  the  designated  location  in  "world  coordinates"  and  did  not  appear  to  move 
when  the  participants  moved  their  heads 

The  IAE  was  updated  to  manage  and  deliver  an  improved  quality  of  signal  through  the 
implementation  of  a  facility  called  "DIS  radios".  Given  the  goal  of  the  program,  the  IAE  is  the  critical 
component  and  was  the  focus  of  the  early  verification  work  in  Task  1.1. 

Shared  development 

The  program  took  advantage  of  work  that  had  been  done  and/or  was  ongoing  in  other  RHCB 
supported  projects.  The  core  of  the  spatializing  audio  software  is  an  open  source  application  from 
NASA,  called  slab3d.  Other  development  programs  by  RHCB  incorporated  this  software  into 
modules  that  could  be  used  within  the  IVP  to  deliver  high  quality  audio. 

To  support  the  use  of  DIS  radios,  the  team  had  to  convert  the  IVP  from  a  HLA  environment  to  a 
homogeneous  DIS  based  system.  This  transition  was  challenging  and  created  a  number  of  technical 
hurdles  that  slowed  the  completion  of  the  technical  work  for  the  experiments.  New  messaging 
types  were  needed  along  with  the  installation  of  DIS  routers,  all  off  which  required  comprehensive 
testing  The  team  appreciates  the  efforts  by  John  Stewart  and  the  support  from  Dr.  Brian  Simpson 
for  making  this  intellectual  property  available.  Similarly,  the  team  thanks  Dr.  Vic  Finomore  for 
providing  access  to  some  DIS  compatible  software  that  enabled  recording  of  communications  for 
subsequent  semantic  analysis. 

An  interactive  series  of  physical  analyses  and  psychophysical  measurements  in  Task  1.1  aided  in  the 
integration  and  debugging  of  the  IAE,  slab3d,  and  DIS  radio  system.  In  addition,  controls  were 
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integrated  into  the  IEC  to  verify  that  the  audio  parameters  were  set  correctly. 

In  addition  to  supporting  person-to-person  voice  communications,  the  IAE  supported  the  delivery  of 
different  sound  streams.  For  Task  3,  the  technical  team  had  to  define  and  develop  an  audio  display 
linked  to  the  "push  to  talk"  control  capability  that  allowed  an  individual  to  do  two  things  with  their 
voice. 

1.  Localize  their  own  voice  so  when  they  spoke  their  partner  would  be  able  to  determine  where 
they  were  speaking  from  in  the  environment  (i.e.,  their  voice  sounded  like  it  was  coming 
from  the  direction  of  their  location).  During  the  design  of  an  earlier  experiment,  PJs 
commented  that  maintaining  an  accurate  sense  of  where  their  team  members  were  was  a 
real  problem.  Task  3,  which  was  done  on  a  one-to-one  basis,  only,  was  the  initial  attempt  to 
assess  the  potential  for  spatialized  audio  to  help  with  this  problem. 

2.  Annotate  or  attach  their  voice  or  an  audio  "tag"  to  an  individual,  location,  or  object  in  the 
environment,  so  that  another  team  member  would  hear  it  coming  from  the  location  of  the 
tagged  object.  This  amounts  to  "audio  pointing"  and  was  assessed  in  Task  3. 

The  challenge  was  to  create  an  audio  display  in  which  there  was  a  readily  perceptible  difference 
between  the  two  signal  type  so  that  an  individual  would  be  able  to  discern  what  they  were  hearing 
and  its  meaning,  i.e.  hearing  a  team  mate's  voice  where  he/she  was  versus  his/her  voice  associated 
with  an  object,  such  as  a  landmark  in  the  environment.  This  capability  was  developed  and 
implemented. 


Task  related  development 
Task  1 

Subtask  1.1  used  the  ALF  simulation  to  evaluate  the  effectiveness  of  the  audio  environment 
at  the  physical  level  plus  procedures  to  make  sure  that  individual  HRTFs  were  correctly 
recorded  and  loaded. 

Task  4 

There  were  no  major  changes  in  the  IAE  for  this  experiment,  but  sound  files  had  to  be 
created  for  every  single  location  of  a  threat  in  3  different  conditions  requiring  two  different 
files: 

•  Spatialized  audio, 

•  Semantic  audio,  i.e.  a  description  of  where  a  particular  threat  was  located-it  was 
time-consuming  and  complex  to  develop  reasonable  descriptions  of  where  snipers 
were.  For  example,  what  is  the  best  way  to  say  that  the  sniper  is  in  the  3rd  building 
over  on  the  4th  floor?  However,  this  was  necessary  to  create  a  baseline  against  which 
to  assess  the  relative  efficacy  of  spatialized  audio,  and 
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•  Mono  files  that  they  were  consistent  in  terms  of  the  presentation  of  information,  i.e. 
a  voice  saying  "Sniper." 


Task  3 

Task  3  involved  the  full  implementation  of  DIS  radios  plus  software  to  handle  multichannel 
communications  with  voice  to  text  translation.  The  audio  display  was  implemented  to 
differentiate  between  localizing  a  person  and  supporting  annotation.  It  should  be  pointed 
out  that  this  annotation  capability  will  become  a  crucial  element  in  experiments  dealing  with 
the  integration  of  command-and-control  capabilities  or  observers  using  other  platforms  over 
the  environment  such  as  UAV  specialists.  When  they  identify  threats  or  other  concerns, 
annotation  may  let  them  improve  how  they  can  effectively  bring  this  information  to  the 
attention  of  troops  on  the  ground. 

The  IAE  had  to  operate  across  the  two  environments  of  the  full  IVP.  Making  sure  that  the 
network  conductivity  and  latency  could  be  managed  so  that  there  were  not  any  overly  long 
delays  in  the  dialog  between  subjects  that  would  feel  unnatural. 

For  Task  3,  the  IAE  recorded  the  conversations  between  subjects.  Given  the  scale  of  the 
experiment,  this  provided  a  large  base  of  data  that  will  be  used  in  future  work  to  understand 
the  effect  of  the  different  communication  modes,  spatial  versus  mono,  in  terms  of  the 
semantic  content  of  the  communications.  This  analysis  could  provide  important  insights  into 
the  effectiveness  of  the  different  audio  modalities;  mono  versus  spatialized,  in  terms  of  its 
effect  on  the  nature  of  the  dialogue  between  subjects.  This  could  provide  the  basis  for 
defining  protocols 

that  could  be  invoked  when  using  spatialized  audio. 

IVP  Data  Base  (IDB) 


Purpose 

Conducting  experiments  in  the  IVP  generates  a  tremendous  amount  of  data,  much  of  it  very  low 
level  in  nature,  such  as  time  and  head  position,  and  a  very  high  frequency  of  collection  (i.e.  at  frame 
rates  (25-50  Hz)  or  cycle  to  cycle  on  a  processor).  The  IDB  has  to  provide  a  database  structure  that 
can  deal  with  the  volume  and  speed  of  the  data,  capture  it  on  a  timely  basis,  perform  additional 
manipulations,  and  support  extensive  subsequent  analysis  to  interpret  the  data.  There  was  not 
adequate  time  or  funding  to  build  an  IDB  with  these  capabilities.  As  previously  noted,  a  distributed 
approach  proved  necessary  with  some  data  being  collected  locally  in  the  IVE  and  other  centrally  in 
the  IEC. 

In  addition,  the  IDB  must  also  capture  and  manage  the  various  DIS  messages  being  sent  within  the 
environment.  A  commercial  application  from  Mak  Technologies,  the  Data  Logger,  was  used  to  do 
this  and  it  supports  the  ability  to  re-create  events  that  have  taken  place  within  the  IVP  based  on  the 
associated  DIS  messages. 
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Technical  development 


Early  in  the  program,  the  team  recognized  that  the  management  of  data  will  ultimately  become  the 
most  significant  component  in  realizing  value  from  the  use  of  virtual  environments,  like  the  IVP  to 
support  assessments  of  technologies  to  improve  human  and  mission  performance.  During  the 
program,  a  great  deal  was  learned  about  the  requirements  for  an  effective  IDB  architecture  and 
implementation,  but  no  substantive  technical  development  was  completed  in  this  area  because  of 
the  other  demands  on  time  and  resources.  The  following  reflect  some  of  what  was  learned. 

1.  Volume  of  data 

The  IVP  generates  significant  volumes  of  data  because  of  the  rate  at  which  data  are 
collected,  frame-to-frame  and/or  processor  cycle  to  cycle  if  necessary,  and  the  diversity  of 
that  data.  Further,  it  was  essential  that  the  collection  of  these  data  took  place  without 
impinging  on  the  real-time  execution  of  the  visual  and  audio  environments  and  any  other 
interactions  that  were  going  on  including  movement,  shooting  etc. 

2.  Decentralized  collection 

To  complete  the  various  tasks,  some  of  the  data  recording  was  implemented  locally  at  the 
data  collection  site  and  was  not  transmitted  to  a  central  storage  site,  as  originally  planed. 
However,  as  bandwidth  increases  and  latency  diminishes,  it  may  be  possible  to  use  a 
centralized  repository.  This  would  eliminate  difficulties  that  arise  when  collecting  data  in 
individual  environments,  including  the  synchronization  of  data  and  its  consolidation  for  the 
analytic  purposes. 

3.  Analytical  complexity 

It  also  became  clear  that  the  subsequent  analytics  were  not  trivial  in  terms  of  the 
calculations  that  needed  to  be  made.  A  comprehensive  IDB  approach  would  define  how  the 
raw  low-level  data  would  be  used  to  create  a  derivative  set  of  data  that  could  then  be  used 
for  analysis,  whether  that  is  related  to  the  semantics  of  voice  communication  or  around 
physical  interactions  and  events. 

One  example  of  the  complexity  of  data  is  trying  to  answer  the  simple  question  of  whether  or 
not  two  individuals  can  see  each  other  and/or  some  3rd  object  in  the  virtual  environment. 
Being  able  to  do  this  in  a  real  time  manner  makes  the  use  of  virtual  environments  very 
compelling  for  evaluation  and  training,  certainly  with  respect  to  the  effect  of  technologies  on 
enhancing  situational  awareness. 

Task  related  development 

Task  4 

Task  4  made  it  clear  that  data  collection  across  the  network  was  going  to  be  problematic  and 
that  there  was  a  need  to  provide  some  level  of  the  data  collection  in  each  location  to 
maintain  the  necessary  synchronicity  and  not  slow  down  the  entire  environment.  As  a  result, 
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the  initial  design  of  having  all  of  the  data  collected  in  the  IEC  was  changed  to  have  most  data 
collected  locally  in  the  IVE. 


Task  3 

Data  collection  for  this  experiment  was  very  different  due  to  the  freedom  the  subject  had  in 
moving  through  the  environment  and  the  quantity  of  audio  data  that  was  collected. 
Understanding  how  spatialized  audio  affected  the  search  process  will  require  more  complex 
analyses  of  the  data,  including  audio  files.  The  data  also  dealt  with  shooter  interactions  with 
hostile  entities  in  the  environment  keeping  track  of  the  number  of  times  individual 
PJs/rescuers  were  hit.  This  is  important  because  an  objective  for  a  PJ/rescuer  is  to  avoid 
getting  injured  and  thereby  needing  to  be  rescued. 


Terrain 


Purpose 

This  defines  the  physical  aspects  of  the  IVP  environment  in  which  the  experiments  occur,  including 
buildings,  doors,  windows,  roadways,  vegetation,  and  their  features.  All  of  these  were  built  using 
the  Presagis'  Creator  application  that  provides  a  suite  of  tools  to  create  objects  that  can  be 
rendered  in  three  dimensions.  This  application  was  the  one  used  to  create  the  simulated  ALF  sphere 
used  in  Task  1  and  the  urban  terrains  used  in  Task  3  and  4. 


Shared  development 
1.  Terrain  tiles 

The  terrain  used  in  previous  experiments  was  limited  in  terms  of  its  physical  dimensions 
(virtual)  and  of  the  variation  that  it  offered  for  the  execution  of  complicated  experiments 
that  required  subjects  to  experience  multiple  trials  in  novel  environments.  Some  of  those 
limitations  and  constraints  were  driven  by  the  nature  of  the  hardware  but  it  became  clear 
from  the  earlier  helicopter  experiment  that  a  more  sophisticated  approach  was  needed. 
Borrowing  from  some  board  games,  the  WSARC  team  proposed  that  terrain  be  developed  in 
"tiles."  These  "tiles"  could  then  be  put  together  so  that  any  edge  in  one  could  be  matched 
up  with  the  edge  of  another,  regardless  of  orientation.  Using  different  "tiles"  in  different 
orientations,  a  large  number  of  varying  terrains  could  be  created.  This  approach  was 
adopted  and  based  on  a  standard  .25  by  .25  kilometer  "tile."  The  work  represented  a  very 
innovative  use  of  the  Creator  environment.  Despite  this,  the  development  effort  to  build  six 
tiles  complete  with  various  types  and  configurations  of  housing  and  urban  scenery  was 
completed  in  three  months  thanks  to  the  dedicated  efforts  of  Andy  Giese,  a  summer  intern 
from  the  University  of  Dayton  and  Chris  Mayhew,  an  intern  working  with  RHCB.  The  six 
"tiles"  were  used  to  create  a  number  of  different  terrains  and  their  processing  was 
optimized  to  sustain  the  necessary  frame  rate  to  maintain  an  acceptable  visual  environment. 
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2.  Object  Hierarchy 


A  key  requirement  that  was  not  understood  at  the  beginning  of  the  development  was  the 
need  to  decompose  the  hierarchies  within  the  building  elements  so  that  things  like  doors 
and  windows  could  be  uniquely  identified.  This  became  important  during  the  development 
of  Task  4  it  became  necessary  to  understand  what  windows,  doors  or  other  features  a 
subject  could  see  from  their  location  in  the  terrain.  As  a  result,  this  detail  was  added  to  the 
Creator  files,  a  task  that  was  both  tedious  and  time  consuming,  but  a  valuable  lesson 
learned. 

By  adding  these  elements  to  the  terrain  definitions,  it  became  possible  to  extract  the  data 
for  the  locations  of  all  components  to  develop  experimental  scenarios  that  insured  two 
things: 

•  that  the  individual  at  a  location  could  see  the  target,  and 

•  that  over  the  experiment  the  targets  were  presented  with  the  necessary  variety  of 
azimuth  and  elevation  to  fully  evaluate  the  effect  of  spatialized  audio  versus  other 
means  of  communication 


3,  Scenario  Generator 

Two  programs  were  developed  to  support  this  design  requirement.  The  first  was  an 
extraction  routine  to  query  the  terrain  database  and  extract  information  on  the  location  of 
all  visible  locations  from  a  subject  location.  The  second  reviewed  these  extracted  locations 
and  selected  the  ones  to  be  used  by  a  sniper  so  that  the  necessary  distribution  of  azimuth 
and  elevation  was  achieved.  Given  the  scope  of  Task  4  to  locate  1,620  such  locations,  it 
would  not  have  been  possible  to  select  sniper  locations  manually,  let  alone  populate  other 
locations  with  ordinary  civilians  to  distract  the  subject.  More  importantly,  this  development 
creates  the  future  possibility  of  a  coordinate  as  opposed  to  entity-based  system  for  defining 
interactions  within  the  IVP.  This  approach  is  more  in  line  with  the  integration  of  sensor 
information  into  the  environment  and  its  communication  from  external  command  centers  or 
UAV  operators. 

4,  Pathing 

Another  enhancement  to  the  terrains  was  to  develop  the  concept  of  "pathing"  through  the 
environment,  defining  the  starts  and  ends,  and  stopping  points  (ambush  locations)  along  the 
way.  This  was  done  for  Task  4  so  that  the  route  traveled  was  controlled  to  ensure  the 
desired  distribution  of  targets  in  space.  This  capability  was  also  used  in  Task  3  looking  at 
different  points  to  start  individuals  from  as  they  were  searching  for  someone  else. 
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Task  related  development 


Task  4 

This  task  was  the  first  to  use  the  "tiles"  for  the  terrain  with  the  extended  hierarchy  to  include 
doors  and  windows.  Completing  this  work  did  delay  the  execution  of  the  experiment  by  over 
2  months.  In  the  course  of  running  the  experiment,  the  use  of  the  "tiles"  proved  to  be  very 
effective  and  allowed  the  team  to  create  a  large  number  of  terrains;  thereby  reducing  the 
likelihood  of  the  subjects  learning  about  the  terrain  they  were  in.  The  process  to  generate 
new  2X2  "tiles"  became  routine,  mixing  and  matching  the  different  individual  "tiles"  to 
create  new  integrated  terrains  as  required. 

With  6  tiles,  the  number  of  possible  combinations  is  very  large  given  that  any  side  can  be 
abutted  to  any  side,  without  additional  development.  Adding  changes  in  elevation  and  other 
components  to  increase  realism  would  be  a  useful  extension  for  a  future  development 
activity  and  a  small  sub-project  created  interiors  in  a  few  buildings  to  demonstrate  how  an 
individual  could  move  inside  and  move  up  and  down  stairs. 

Task  3 

The  experiment  took  advantage  of  the  "tiled"  terrain,  but  required  a  far  larger  number  of 
unique  "tile"  combinations.  Again,  this  was  done  to  avoid  learning.  Different  starting  points 
for  searches  were  implemented  and  to  evaluate  the  impact  of  landmarks  on  the  task, 
terrains  were  created  with  and  without  landmarks  using  a  toggling  feature  implemented 
within  the  "tiles." 

Artificial  Intelligence  (Al) 


Purpose 

The  two  aspects  of  using  artificial  intelligence  components  within  the  IVP  are: 

1.  Dealing  with  the  simulation  of  events  based  on  other  events  and  interactions  so  that  certain 
actions  unfold  as  the  subject  moves  in  the  environment  or  interacts  with  it,  and 

2.  Introducing  digital  entities  in  the  environment  that  can  be  made  to  behave  in  certain  ways 
depending  on  circumstances.  For  example,  the  use  of  Al  might  allow  an  entity  to  see  a 
dismounted  PJ  and  either  approach  to  attack  or  run  away  and  hide.  These  behaviors  have 
become  increasingly  sophisticated. 

The  treatment  element  of  the  architecture  defines  values  of  the  variables  specified  by  the  design  of 
a  particular  experiment.  These  vary  significantly  based  on  the  experimental  scenarios  and  may 
include:  time  of  day  within  the  virtual  environment,  starting  point  within  a  terrain,  location  of 
waypoints,  partnering  teams,  speed  of  movement,  etc. 
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Shared  development 


All  of  the  experiments  required  the  execution  of  simulated  events.  In  Task  1,  these  were  the 
simulation  or  emulation  of  the  normal  localization  tasks  performed  in  the  ALF.  Task  3  integrated  un¬ 
tethered  movement  by  subjects  through  a  terrain  starting  from  different  points  and  also  meeting 
digital  individuals  (avatars)  with  certain  types  of  preprogrammed  behaviors  for  interacting  with  both 
PJ  and  the  target  subjects. 

Task  related  development 

Task  4 

There  was  no  significant  Al  component  for  this  task. 

Task  3 

For  this  program,  this  task  was  the  only  one  in  which  avatars  with  behaviors  were  included. 
Their  behaviors  were  either  passive,  i.e.  simply  there  as  distractions,  or  as  hostiles 
demonstrating  those  hostile  intents  by  engaging  in  firefights  with  PJs  or  threatening  the 
target  subject  to  force  him/her  to  keep  moving.  As  a  result,  the  Al  for  this  experiment  was 
more  complex  to  support  these  interactions,  but  made  easier  because  the  subjects  were 
able  to  move  freely  in  the  environment  not  through  some  preprogrammed  set  of  pathways. 

Team  members  in  close  proximity  appeared  to  each  other  in  the  form  of  avatars.  A  direction 
indicator  was  present  in  the  center  of  the  base  of  all  walls,  displaying  primary  and  secondary 
headings  (N,  NW,  W,  SW,  etc.).  Participants  used  a  wand  that  projected  a  virtual  line 
extending  from  the  tip  of  the  wand  into  infinity.  This  line  functioned  as  a  visual 
representation  of  the  wand  target. 

Four  thumb-accessible  buttons  on  the  wand  allowed  the  participants  to  interact  with  the 
simulation  environment.  Of  the  other  four  buttons,  one  represented  a  push-to-talk  function 
by  which  participants  could  communicate  with  one  another  in  all  conditions.  The  other  three 
buttons  pertained  to  the  spatial  audio  condition  discussed  further  below.  An  additional 
trigger  functioned  as  a  virtual  gun  for  the  PJs  to  use  against  enemies. 


Treatments 


Purpose 

As  mentioned  the  range  of  treatments  possible  within  the  IVP  can  be  highly  varied  and  controlled 
very  precisely.  These  now  include  the  ability  to 

•  Mix  and  match  different  terrain  "tiles" 

•  Execute  from  different  starting  points 

•  Follow  different  paths 
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•  Run  exercises  at  different  times  in  the  day 

•  Involve  digital  entities  creating  different  interactions 

•  Engage  different  subjects  and/or  teams 

These  have  to  be  defined  and  managed  to  support  a  particular  experimental  design.  As  a  result,  the 
creation  of  a  system  to  define  and  manage  the  treatments  down  to  blocks  of  experiments,  to  the 
teams  within  the  blocks,  and  so  on  was  another  significant  deliverable  from  the  program.  Ensuring 
that  these  elements  came  together  appropriately  was  a  very  important  aspect  of  the  engagement 
between  the  researchers  and  the  technical  team. 

Task  related  development 

Task  4 

Task  4  involved  6  subjects  each  moving  through  9  different  terrain  each  with  30  possible 
locations  of  which  10  were  selected  for  each  type  of  audio  display;  spatialized,  semantic,  and 
mono.  It  was  a  large  experiment  with  a  complex  design  in  terms  of  the  application  of  specific 
treatments.  The  development  of  the  XML  treatments  file  was  essential  to  the  successful 
execution  of  the  project  and  its  management.  Without  this  and  the  IEC,  it  is  unlikely  this  task 
would  have  been  completed  considering  that  for  each  of  the  270  locations  that  a  subject  was 
exposed  to  required  a  calculation  to  make  sure  that  they  could  see  the  target  and  that  over 
the  course  of  the  trials  for  a  particular  audio  treatment;  spatial,  semantic  or  monaural  there 
was  an  even  distribution  in  terms  of  azimuth  and  elevations.  This  is  not  trivial  and  required  a 
great  deal  of  effort  to  develop  these  files  and  then  to  test  all  of  the  locations  in  all  of  the 
terrains  on  all  paths  to  make  sure  that  they  all  functioned  correctly,  i.e.  that  the  vehicle 
stopped  at  the  appropriate  point,  that  the  targets  presented  themselves  appropriately,  and 
again  that  at  the  end  of  this  experiment  that  an  even  distribution  of  locations  was  achieved. 

Task  3 

The  treatments  had  to  mitigate  learning  and  also  support  the  exploration  of  a  number  of 
different  options  with  respect  to  the  use  of  audio  annotation  (versus  non-annotation)  and  of 
landmarks,  based  on  where  subjects  started.  Because  the  subjects  were  free  to  move 
anywhere  to  find  each  other,  the  treatments  were  constructed  to  create  the  maximum 
amount  of  flexibility  and  variation  across  the  entire  set  of  experiments.  This  approach  would 
level  out  any  differences  in  terrains,  for  example  some  terrains  were  more  difficult  to 
navigate  and  or  certain  hostile  interactions  were  harder  to  deal  with. 


4.4  Results,  Conclusions  And  Recommendations 

This  section  documents  the  results  of  the  technical  development  effort  during  the  program,  the 
conclusions  or  "lessons  learned"  from  that  work,  and  the  recommendation  for  future  priorities 
and/or  directions.  There  will  be  a  similar  section  for  each  of  the  task  write-ups  and  one  for  the 
overall  program. 
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Results 

There  were  three  key  results  from  the  technical  development  activities  completed  in  the  course  of 
the  program. 

1.  Implementation  of  significant  enhancement  to  IVP 


a.  Transition  to  an  all  DIS  environment 

This  was  done  to  ensure  compatibility  with  other  AF  distributed  simulations  and  virtual 
environments  eliminating  the  HLA  components  and  also  to  enhance  the  audio  quality  of  the 
environment.  The  latter  was  through  the  implementation  of  DIS  radios  to  generate  audio 
sounds  rather  than  Team  Speak  which  limited  the  frequency  range  available,  particularly  at 
the  higher  ranges  which  are  important  for  elevation  discrimination.  This  conversion  was 
time  consuming,  but  the  impact  on  audio  quality  was  evident.  As  a  result  of  the  change,  the 
installation  of  DIS  routers  to  move  message  between  the  two  IVP  environments  was 
necessary  and  also  the  development  of  special  DIS  message  types  replacing  previously  used 
HLA  ones. 

b.  Development  of  experiment  controller 

In  2007,  the  recommendation  was  made  to  implement  an  experiment  controller  to  manage 
the  loading  of  configuration  files  for  a  particular  experimental  trial,  the  synchronization  of 
the  startup  for  all  applications  in  one  or  both  virtual  environments,  the  execution  of  the 
trials  themselves  in  terms  of  starting  and  stopping  certain  events  or  initiating  activities 
within  the  trial,  and  the  shutdown  of  the  trial  and  the  subsequent  closing  of  all  files  and 
applications  including  any  of  data  files  used  to  collect  information  from  the  experiment.  As 
noted,  two  different  versions  were  deployed  but  these  should  be  integrated  and  that  is  a 
recommendation  going  forward. 

c.  Implementation  of  Terrain  "Tiles" 

The  technical  team  pioneered  the  development  of  terrain  "tiles",  which  could  be 
manipulated  to  create  a  very  wide  variety  of  subject  environments  for  experiments.  In 
addition,  the  size  of  the  terrain  in  which  the  experiments  could  take  place  was  dramatically 
enhanced  from  a  few  hundred  meters  to  .5  kilometers  along  the  edges  and  close  to  three 
quarters  of  the  kilometer  on  the  diagonal.  This  supported  experiments  of  much  longer 
duration  than  was  previously  possible  and  as  a  result  the  introduction  of  more  complex 
elements  into  each  of  the  experimental  design  scenarios.  In  addition,  the  creation  of 
hierarchies  within  the  rendering  software  (Creator)  provided  the  capability  to  do  very 
detailed  analysis  on  "field  of  view"  and  "line  of  sight"  defining  occlusions  without  the  need 
to  introduce  additional  entities.  This  capability  will  become  particularly  important  as  the  IVP 
is  further  integrated  with  other  systems  such  as  command-and-control  with  ISR  capabilities 
from  UAVs,  etc. 
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3.  Development  of  three  unique  experimental  environments 

The  virtual  environments  experienced  by  subjects  in  Tasks  1,  3  and  4  were  dramatically 
different.  Task  1  implemented  a  virtual  analog  to  the  ALF  sphere  environment  including  the 
ability  to  emulate  the  types  of  experiments  conducted  in  that  environment.  Although  Tasks  3 
and  4  used  the  urban  terrain  developed  using  "tiles",  the  ways  in  which  the  subjects 
interacted  with  them  were  vastly  different.  In  Task  4,  the  subject  rode  in  a  Humvee  traveling 
in  a  convoy  through  the  urban  environment,  and  in  the  other  the  subjects  were  able  to  move 
freely  within  the  environment  as  they  searched  for  each  other 

4.  Development  of  compelling  and  highly  usable  experimental  environments 

The  subjects  used  during  the  course  of  the  various  experiments  were  untrained  in  terms  of 
the  specific  missions  that  they  were  asked  to  perform.  They  were  also  inexperienced  in 
working  in  a  virtual  environment  and  using  the  interactive  devices.  It  was  found  that  with 
relatively  little  training,  subjects  were  able  to  operate  effectively  in  the  technical 
environment,  to  move  through  a  virtual  space  and  to  execute  the  assigned  tasks.  This  is 
indicative  of  the  real  advantage  of  using  virtual  environments  for  not  only  the  evaluation  of 
technologies,  but  also  the  completion  of  education  and  training  activities  on  an  accelerated 
and  very  cost-effective  basis. 

Conclusions  (Lessons  Learned) 

Despite  the  work  that  had  been  done  using  the  IVP  since  2007,  a  great  deal  was  learned  in  the 
execution  of  this  program  in  terms  of  the  challenges  of  working  in  virtual  environments  and 
developing  effective  and  meaningful  experimental  scenarios. 

1.  Technical  workload  and  resources 

The  technical  workload  was  underestimated.  The  experiments  that  were  planned  and 
conducted  became  increasingly  complex  and  as  a  result  the  environment  and  all  of  the 
adjunct  components  became  correspondingly  complex.  It  is  unlikely  that  these  requirements 
will  diminish  for  future  experiments,  rather  the  IVP  will  need  additional,  more  sophisticated 
capabilities.  It  is  also  clear  that  the  pool  of  individuals  capable  of  performing  this  work  is  very 
limited  and  aggressive  efforts  must  be  made  to  expand  it. 


2.  Level  of  technical  readiness 

The  IVP  was  less  reliable  than  expected  at  the  time  that  the  SOW  was  developed  in  concert 
with  the  WSU  team.  As  noted,  the  results  of  the  experiment  conducted  in  later  2009 
immediately  prior  to  this  program  pointed  to  a  need  to  significantly  improve  the  quality  of 
validation  and  verification  procedures  for  the  IVP.  These  are  essential  to  ensure  that  the 
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environment  is  performing  correctly  in  terms  of  the  basic  elements  of  hardware  and 
software,  particularly  those  related  to  the  audio  function,  which  was  the  focus  of  the 
experimental  activities.  As  a  result,  a  significant  portion  of  the  effort  on  Task  1  was  devoted 
to  full  end-to-  end  verification  and  validation  of  the  audio  environment  using  the  ALF  facility 
to  actually  physically  measure  sounds  and  make  sure  that  they  correlated  correctly  to  what 
was  being  heard  in  the  virtual  environment.  Going  forward  it  will  become  increasingly 
important  to  repeat  these  validation  and  verification  procedures  including  the  calibration  of 
subject  and  their  HRTF  files  to  ensure  that  they  are  correct  and  are  being  loaded  properly 
before  any  experiment  is  started. 

3.  Amount  of  testing 

This  was  under-estimated,  particularly  what  was  required  to  ensure  that  the  environment 
used  for  a  particular  experimental  trial  was  performing  correctly.  Over  above  the  normal 
testing  of  the  hardware,  software  and  communications,  the  technical  team  had  to  verify  that 
all  of  the  experimental  sequences  a  subject  would  experience  performed  correctly.  For 
example,  if  a  subject  had  to  start  a  particular  exercise  in  a  specific  location  in  the 
environment,  then  the  test  had  to  be  conducted  to  make  sure  that  the  subject  in  fact  could 
start  there  and  that  the  experience  they  had  from  that  point  forward  was  consistent  with 
the  design  of  the  experiment.  This  meant  testing  all  the  possible  combinations  of  treatments 
within  a  particular  experiment  and  for  Task  4  this  involved  the  technical  team  reviewing  over 
1,000  individual  combinations  before  subjects  were  permitted  to  use  the  environment. 

4.  Data  management 

The  purpose  of  building  the  IVP  was  to  conduct  experiments  and  collect  the  detailed  data  to 
assess  the  performance  of  spatialized  audio.  This  requires  a  sophisticated  data  collection 
capability  and  while  some  suitable  intermediate  solutions  were  found  plus  the  use  of  the 
Mak  Data  Logger  to  replay  scenarios,  it  became  apparent  that  a  more  comprehensive 
scheme  for  the  management  of  data  within  the  IVP  is  a  priority. 

5.  Adaptable  environment 

The  IVP  is  a  very  compelling  two  location,  virtual  environment  in  which  to  conduct 
experiments  to  evaluate  the  effectiveness  of  audio  display  systems.  However,  it  is  also  clear 
that  it  could  be  adapted  to  evaluate  other  types  of  technology  simply  by  modifications  to  the 
various  software  elements. 


6.  Process  management 

The  technical  management  processes  need  to  be  improved,  starting  with  a  more  rigorous 
environment  for  software  development,  more  effectively  linking  design  and  development  to 
the  experimental  design.  While  designs  were  eventually  developed  that  supported  the 
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experiments  to  assess  human  performance  and  the  effect  of  audio  display  technologies,  the 
process  for  developing  these  was  arduous.  As  expected,  it  was  highly  collaborative  and 
iterative,  but  given  that  the  ability  to  execute  experiments  is  heavily  dependent  on  the 
technical  environment  and  development,  better  processes  for  eliciting  the  experimental 
needs  and  translating  these  into  highly  reliable  applications  needs  to  be  implemented. 

7.  Creative  solutions 

The  use  of  the  terrain  "tiles"  is  only  one  example  of  the  creative  technical  solutions 
developed  during  the  course  of  the  program.  Others  include  the  IEC,  the  "scenario 
generator"  and  the  ALF  simulator.  All  of  these  will  deliver  continuing  value  in  future 
experiments  involving  the  IVP.  For  example,  the  "tile"  based  terrain  supports  a  coordinate 
based  view  of  the  environment  that  will  be  extremely  important  as  additional  experiments 
may  introduce  components  like  helicopters  or  UAVs  and  communications  become 
dependent  on  resolving  questions  about  joint  visibility  in  urban  environments. 

8.  Performance 

Overall,  the  IVP  performed  well.  The  processors  being  used  have  adequate  horsepower  for 
the  nature  of  the  trials  that  were  undertaken  and  it  is  anticipated  that  this  will  be  true  for  at 
least  the  next  few  years.  The  projection  environments  will  be  viable  and  work  effectively  for 
at  least  this  long. 

Recommendations 

From  a  technical  perspective  there  are  there  specific  recommendations  in  terms  of  priorities  for  the 

next  round  of  developments  in  the  use  of  the  IVP. 

1.  Completion  of  IEC 

Having  a  single,  user  friendly  IEC  to  control  the  IVP  and  the  execution  of  experiments 
execution  is  essential.  The  purpose  of  developing  the  IEC  was  to  allow  less  technical  users  to 
set  up  the  correct  environments  to  run  experiments  and  to  be  able  to  manage  the  operation 
of  the  IVP  successfully  during  their  research.  There  are  currently  two  versions  and  these 
need  to  be  merged  into  a  single  GUI-based  design.  This  is  critically  important  to  improve 
access  to  the  IVP  as  a  virtual  environment  for  conducting  human  performance  related 
experiments  and  as  such  should  be  the  first  priority  for  the  next  round  of  technical 
development  to  be  done  on  the  system. 

2.  Enhanced  IDB 

The  current  database  is  not  adequate,  particularly  with  the  addition  of  audio  files  and  more 
sophisticated  team-based  interactions  with  more  entities  involved.  The  Mak  Data  Logger, 
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which  captures  the  DIS  messaging  is  adequate  to  provide  context  but  a  far  more 
sophisticated  system  for  retrieving  and  analyzing  experimental  data  will  be  needed  as  the 
IVP  is  used  in  the  future. 

3.  Ambient  noise 

The  audio  environment  (IAE)  needs  to  be  enhanced  to  introduce  accurately  spatialized 
ambient  noise  in  the  IVP.  Task  2  in  the  SOW  was  to  investigate  this  particular  area  but  was 
not  started  because  of  the  outstanding  technical  issues  from  previous  research.  However, 
from  the  work  that  was  done,  it  is  important  to  be  able  to  understand  and  differentiate 
different  types  of  noises  within  the  environment.  In  fact,  this  will  be  essential  if  a  spatialized 
audio  capability  is  ever  going  to  be  delivered  to  the  field.  It  is  clear  that  PJs  rely  on  their 
sense  of  hearing.  Even  if  having  spatialized  audio  displays  significantly  improves  their 
performance  in  navigation,  threat  mitigation  and  situational  awareness,  it  must  not  interfere 
with  other  audio  inputs,  such  as  the  opening  of  a  door,  that  they  must  also  hear  and  respond 
to  appropriately. 

4.  Terrain  Enhancement 

The  implementation  of  the  tiling  concept  significantly  enhanced  the  functionality  of  the  IVP 
and  the  ability  to  use  it  for  more  sophisticated  experiments.  The  next  round  of  investment 
should  build  on  this  by  adding  more  complex  topographies  and  allowing  subjects  to  enter 
and  move  around  in  buildings. 
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5.0  SECTION  5:  EXPERIMENTAL  TASK  REPORTS 


5.1  Introduction 

The  following  three  sub-sections  are  the  experimental  reports  primarily  prepared  by  the  WSU 
research  team  headed  up  by  Dr.  Gilkey  as  the  Principal  Investigator.  The  reports  included  for  Tasks  3 
and  4  are  being  submitted  to  a  conference  and  so  their  format  is  consistent  with  those 
requirements.  This  seemed  a  more  expedient  and  cost  effective  approach.  The  report  for  Task  1 
follows  a  similar  format,  but  is  not  intended  for  a  conference  or  publication. 
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5.2  Section  5.2  Task  1:  The  Influence  Of  Listening  Environment  On  Sound 
Localization 

We  can  expect  that  the  effectiveness  of  a  3D  audio  display  will  depend  to  a  large  degree  on 
the  ability  of  that  display  to  render  sounds  that  appear  to  arise  from  the  intended  spatial  location. 
The  goal  of  this  study  was  to  compare  localization  performance  in  the  IVP  to  a  "known"  standard. 
Specifically,  a  number  of  studies  have  demonstrated  that  very  good  localization  performance  can  be 
realized  in  the  Auditory  Localization  Facility  (ALF)  of  RHCB  at  WPAFB  (e.g.,  Simpson,  2002,  Simpson, 
2012).  There  are  at  least  three  reasons  to  expect  good  localization  of  live  sounds  in  this  facility. 

First,  ALF  is  housed  in  a  large  double-walled  anechoic  chamber  and  thus  provides  a  very  quiet 
listening  environment.  Second,  the  facility  is  able  to  produce  high-fidelity,  broad-bandwidth  sounds 
(.2  to  14  khz).  Third,  speaker  specific  preprocessing  of  sounds  ensures  that  the  system  response  is 
flat  and  the  same  independent  of  speaker  location. 

Significantly,  localization  performance  with  virtual  sounds  presented  through  headphones 
with  SLAB  (Sound  Lab,  Wenzel,  Miller,  &  Abel,  2000;  Miller  &  Wenzel,  2002)  when  heard  in  ALF  is 
nearly  as  good  as  when  live  sounds  are  presented  from  the  loudspeakers  (e.g.,  Simpson,  2002; 
Brungart,  Romigh,  &  Simpson,  2011).  Listeners  report  that  virtual  sounds  are  hard  to  distinguish 
from  real  sounds  and  appear  to  arise  from  the  speakers  themselves  rather  than  the  headphones. 

Even  though  the  physical  sounds  produced  by  SLAB  should  be  identical  whether  they  are 
presented  in  ALF,  VERITAS,  or  some  other  listening  environment,  there  are  several  reasons  to 
expect  the  perception  of  those  sounds  to  be  particularly  good  in  ALF.  First,  as  mentioned,  ALF  is  a 
very  quiet  listening  environment.  The  ambient  noise  level  in  ALF  is  about  44  dBC,  whereas  the 
ambient  noise  level  in  VERITAS  is  about  66  dBC.  Previous  research  (Good  and  Gilkey,  1996;  Lorenzi, 
Gatehouse,  &  Lever, 1999;  Simpson,  2011)  indicates  that  localization  performance  is  severely 
impacted  by  noise,  particularly  with  regard  to  front/back  discrimination.  Second,  the  head-related 
transfer  functions  (HRTFs,  filters  that  reproduce  the  spectral  and  temporal  modifications  that  occur 
for  a  live  sound  as  it  travels  from  a  particular  source  location  to  the  eardrums  of  a  particular 
listener)  that  we  use  to  generate  virtual  sounds  with  SLAB  are  recorded  in  ALF.  To  the  degree  that 
high-fidelity  signal  processing  is  maintained,  the  sounds  that  reach  the  listener's  ears  for  live  and 
virtual  presentations  in  ALF  should  be  identical.  That  is,  virtual  stimuli  should  sound  "just  like"  live 
stimuli.  In  contrast  to  the  fairly  anechoic  listening  environment  of  ALF,  VERITAS  is  very  reverberant, 
so  live  stimuli  presented  in  VERITAS  would  sound  quite  different  from  the  virtual  stimuli.  If  subjects 
try  to  localize  the  virtual  stimuli  to  the  location  that  a  similar  sounding  live  stimulus  could  have 
come,  we  might  expect  to  see  much  poorer  localization  performance.  Third,  Gilkey,  Simpson,  and 
Weisenberger  (2001)  showed  that  binaural  recordings  "sound  better"  when  heard  in  the  same 
location  where  the  recordings  were  made.  While  this  appears  to  be  related  to  the  acoustic 
mismatch  described  above,  the  visual  mismatch  also  seems  to  be  a  factor.  That  is,  it  is  as  if  the 
listener  uses  the  visual  scene  to  determine  what  the  stimuli  should  "sound  like"  in  that  environment 
and  judges  them  as  less  realistic  when  they  sound  different  than  expected.  So,  for  a  number  of 
reasons  we  expected  localization  performance  in  VERITAS  to  be  worse  than  in  ALF.  Task  1  was 
designed  to  measure  that  difference. 
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Method 


Listeners.  A  total  of  4  male  and  1  female  normal  hearing  listeners,  ranging  from  20-26  years 
of  age,  participated  in  the  study.  The  listeners  were  part  of  the  paid  listener  panel  maintained  by 
the  Battlespace  Acoustics  Branch  at  Wright-Patterson  Air  Force  Base.  Invidualized  HRTFs  were 
recorded  for  each  participant  in  the  ALF,  with  corrections  for  the  HD280  headphones  used  in  this 
study. 


Apparatus. 

ALF.  When  in  ALF  the  listeners  straddled  a  bench  with  their  head  at  the  center  of  a  4.3-m 
geodesic  sphere  with  277  Bose  11-cm  full-range  speaker  drivers  mounted  its  surface  (see  Figure  4). 
The  sphere  is  housed  in  a  large  double-walled  anechoic  chamber.  Custom-built  high-power 
switching  hardware  (Winntech)  can  route  up  to  15  signals  to  any  or  all  of  the  277  speakers. 
However,  for  this  study  all  sounds  where  rendered  at  the  appropriate  speaker  locations  using 
Slab3d  (version  6.0.1)  An  Intersense  IS-900  tracking  device  monitors  the  position  of  the  subject's 
head  and  of  a  handheld  wand  that  is  used  as  a  response  device  (as  described  below).  A  cluster  of 
LEDs  is  mounted  in  the  center  of  each  speaker,  which  can  be  used  to  provide  cuing  and  feedback  to 
the  subject,  as  described  below. 


Figure  4.  The  Auditory  Localization  Facility  at  Wright-Patterson  AFB. 

VERITAS.  When  in  VERITAS  (see  description  in  Section  4.2),  the  listeners  sat  in  a  swivel  chair 
with  their  heads  in  the  center  of  a  visual  simulation  of  the  ALF  sphere.  The  simulation  consisted  of 
3D  models  of  the  speakers  made  to  appear  11-cm  in  diameter  at  locations  corresponding  to  the 
speakers  in  ALF  (the  struts  of  the  ALF  sphere  were  not  rendered).  Because  VERITAS  only  has  4  walls 
and  a  floor  and  does  not  have  a  top  projection  surface,  speaker  locations  above  45°  elevation  were 
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not  rendered.  Four  red  dots  in  the  center  of  each  simulated  speaker  mimicked  the  LEDs  in  the  ALF 
sphere.  They  could  be  brightened  and  dimed  under  program  control  and  were  used  to  provide 
cuing  and  feedback  (as  described  below).  The  IAE  with  Slab3d  (version  6.5.0)  was  used  to  render 
sounds  at  the  appropriate  speaker  locations.  VERITAS'  Intersense  IS-900  tracking  system  monitored 
the  position  of  the  subject's  head  and  of  a  handheld  wand  that  was  used  as  a  response  device 
(described  below). 

Procedure.  The  procedures  used  were  similar  to  those  used  in  numerous  previous 
experiments  conducted  in  ALF;  however,  some  slight  modifications  were  needed  to  accommodate 
the  differences  between  ALF  and  VERITAS. 

Stimuli.  The  target  to  be  localized  in  ALF  was  a  250-ms  burst  of  white  noise  that  was  square 
gated  on  and  off  and  presented  at  approximately  60  dB  SPL  (measured  at  the  center  of  the  sphere 
with  no  listener  present).  In  VERITAS,  the  target  was  a  100-ms  white  noise  with  5-ms  Blackman 
onset  and  offset  ramps.  Listeners  were  allowed  to  adjust  the  target  to  a  comfortable  listening  level. 
The  same  set  of  speaker  locations  was  used  in  both  ALF  and  VERITAS.  Because  some  speaker 
locations  below  -45°  elevation  are  partially  blocked  by  obstructions  in  ALF,  speakers  below  -45°  are 
typically  not  used.  As  mentioned,  because  there  is  no  top  screen  in  VERITAS  it  was  not  possible  to 
visually  render  speaker  locations  above  +45°  elevation.  Therefore,  the  speaker  locations  used  in 
this  study  were  restricted  to  the  200  speakers  between  -45°  and  +45°  elevation.  All  sounds  were 
virtual  and  slab3d  was  used  to  spatialize  the  sounds  in  both  locations. 

Trial  sequence  in  ALF.  In  ALF,  the  subjects  straddled  a  bench  that  was  mounted  on  a 
platform  in  the  center  of  the  sphere.  To  begin  each  trial  the  subjects  had  to  position  their  head  in 
the  center  of  the  sphere  and  orient  toward  the  speaker  at  0°  azimuth  and  0°  elevation;  the  LED 
cluster  on  that  speaker  indicated  when  this  position  was  obtained.  Once  the  LED  cluster  was  lit,  the 
subjects  could  press  a  button  on  the  wand  when  they  were  ready  for  a  trial  to  begin.  The  stimulus 
was  then  presented.  After  the  stimulus  was  presented  the  subjects  turned  toward  the  perceived 
location  of  the  sound  and  pointed  with  the  wand  at  the  speaker  that  they  believed  was  in  the 
direction  of  the  sound.  The  speaker  they  were  pointing  to  was  indicated  by  illuminating  the  cluster 
of  LEDs  on  that  speaker.  When  the  intended  speaker  was  indicated,  they  pressed  a  button  on  the 
wand  to  register  their  response.  After  they  responded,  feedback  was  provided  by  illuminating  the 
LED  cluster  on  correct  speaker.  The  subjects  pressed  a  button  to  acknowledge  the  feedback  and 
oriented  to  the  front  speaker  to  get  ready  for  the  next  trial. 

Trial  sequence  in  VERITAS.  The  trial  sequence  in  VERITAS  was  similar  to  that  in  ALF. 

However  the  subjects  were  seated  in  a  chair  that  swiveled  allowing  them  to  turn  to  face  any 
speaker.  The  head  tracker  indicated  when  the  subject  was  oriented  toward  the  virtual  speaker  at  0° 
azimuth  and  0°  elevation;  the  VERITAS  ALF  model  included  a  small  "F"  under  the  0°  azimuth  and  0° 
elevation  speaker  and  "B"  under  the  180°  azimuth  and  0°  elevation  speaker  to  help  avoid 
disorientation  about  which  was  the  front  or  back  speakers.  As  in  ALF,  once  the  correct  position  was 
obtained,  the  subjects  could  press  a  button  on  the  wand  to  initiate  a  trial.  After  the  stimulus  was 
presented,  they  responded  by  pointing  the  wand  at  the  desired  speaker;  the  selected  speaker  was 
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indicated  by  brightening  the  "LEDs"  on  that  speaker.  When  the  intended  speaker  was  selected,  the 
subject  pushed  a  button  on  the  wand  to  register  their  response.  After  they  responded,  feedback 
was  provided  by  brightening  the  LED  cluster  on  correct  speaker.  The  subjects  pressed  a  button  to 
acknowledge  the  feedback  and  oriented  to  the  front  speaker  to  get  ready  for  the  next  trial. 

Block  structure.  Data  in  both  locations  were  collected  in  eight  50-trial  blocks  for  each 
subject.  Each  of  the  200  speaker  locations  was  presented  twice  for  a  total  of  400  trials  per  subject 
per  location. 

Results  and  Discussion 

Initial  psychophysical  measurements  obtained  under  this  project  were  not  as  expected  and 
provided  the  motivation  for  physical  measurements  of  the  stimuli  presented  at  the  two  locations. 
These  analyses  revealed  several  hardware  and  software  bugs  in  the  slab3d  implementation,  the 
individualized  HRTFs,  the  IAE  implementation,  and  the  physical  measurement  procedure. 

The  psychophysical  results  after  addressing  these  problems  are  shown  in  Figures  5  to  9. 

Each  figure  shows  the  results  for  one  of  the  five  subjects.  The  top  row  in  each  figure  shows  the  data 
from  ALF  and  the  bottom  row  shows  the  data  from  VERITAS.  The  data  are  plotted  using  the  3-pole 
coordinate  system  of  Wightman  and  Kistler  (1997).  Each  panel  shows  a  scatter  plot  of  the  response 
coordinate  as  a  function  of  the  stimulus  coordinate  for  the  400  individual  trials.  The  left-hand  panel 
in  each  row  shows  the  Left/Right  coordinates  (the  Left/Right  coordinate  of  a  location  is  the  angle 
between  a  vector  from  the  center  of  the  head  to  that  location  and  the  median  plane).  The  middle 
panel  shows  the  Front/Back  coordinates  (the  Front/Back  coordinate  of  a  location  is  the  angle 
between  a  vector  from  the  center  of  the  head  to  that  location  and  the  frontal  plane).  The  right-hand 
panel  shows  the  Up/Down  coordinates  (the  Up/Down  coordinate  of  a  location  is  the  angle  between 
a  vector  from  the  center  of  the  head  to  that  location  and  the  horizontal  plane).  The  rms  error 
between  the  stimulus  coordinate  and  the  response  coordinate  is  shown  above  each  panel. 
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Figure  5.  Results  from  ALF  (top  row)  and  VERITAS  (bottom  row)  for  Subject  P1379.  Results 
from  ALF  (top  row)  and  VERITAS  (bottom  row)  for  Subject  P1379.  In  each  row,  left,  middle, 
and  right  panels  show  the  Left/Right,  Front/back,  and  Up/Down  coordinates,  respectively. 
Within  each  panel,  each  data  point  shows  the  response  coordinate  for  each  individual  trial 
plotted  function  of  the  coordinate  of  rendered  stimulus  on  that  trial. 
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Figure  6.  Results  for  Subject  P1401  are  plotted  in  the  same  manner  as  Figure  5. 
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Figure  7.  Results  for  Subject  P1442  are  plotted  in  the  same  manner  as  Figure  6. 
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Figure  8.  Results  for  Subject  P1447  are  plotted  in  the  same  manner  as  Figure  6. 
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Figure  9.  Results  for  Subject  P1481  are  plotted  in  the  same  manner  as  Figure  6. 

As  expected,  localization  judgments  in  VERITAS  were  less  accurate  than  those  in  ALF. 
However,  the  magnitude  of  these  differences  was  larger  than  we  anticipated.  In  part,  this  is 
because  performance  in  ALF  was  very  good.  Left/Right  errors  in  ALF  ranged  from  7°  to  14°. 

Although  Left/Right  errors  in  VERTAS  were  poorer,  they  were  still  fairly  low  14°  to  21°.  On  the 
other  hand,  Front/Back  errors  in  VERITAS  were  much  larger,  exceeding  40°  for  most  subjects.  This  is 
largely  due  to  the  substantial  number  of  back  to  front  reversals.  Indeed  for  subject  P1379  most 
sounds  presented  in  the  rear  hemisphere  were  reported  to  be  in  the  front  hemisphere.  The  picture 
in  the  Up/Down  dimension  is  less  clear  because  of  the  limited  range  of  elevations  considered.  That 
said,  performance  was  substantially  worse  in  VERITAS. 

The  reasons  for  the  surprisingly  poor  performance  in  VERITAS  are  uncertain.  As  mention  in 
the  introduction,  differences  in  ambient  noise  levels,  reverberation,  and/or  visual  environment 
could  be  the  cause.  It  is  also  possible  that  the  different  versions  of  slab3d  or  differences  in  the 
duration,  level,  and/or  temporal  envelope  of  the  stimuli  between  ALF  and  VEITAS  could  have 
contributed;  but  we  would  expect  the  impact  of  these  differences  to  be  small.  Finally,  it  is  possible 
that  additional  bugs  in  the  IAE,  slab3d  or  possibly  the  ALF  software,  could  still  be  present.  Limited 
time  and  funding  did  not  allow  for  further  examination  of  these  issues.  Clearly,  additional  effort  is 
needed  going  forward  to  identify  and  resolve  these  questions. 

References 

Brungart,  D.S.,  Romigh,  G.,  &  Simpson,  B.D.  (2011).  Rapid  collection  of  head  related  transfer 

functions  and  comparison  to  free-field  listening.  In  Y.  Suzuki,  D.  Brungart,  Y.  Iwaya,  K.  Lida,  D. 
Cabrera,  &  H.  Kato  (Eds.),  Principles  and  applications  of  spatial  hearing  (pp.  139-148). 
Singapore:  World  Scientific. 


48 

Distribution  A:  Approved  for  public  release;  distribution  unlimited.  88ABW  Cleared  05/29/2012;  88ABW-201 2-3075. 


Gilkey,  R.  H.,  Simpson,  B.D.,  &  Weisenberger,  J.  M.  (2001).  Creating  auditory  presence.  Usability 
Evaluation  and  Interface  design:  Cognitive  Engineering ,  Intelligent  Agents,  and  Virtual  Reality, 
Volume  I  of  the  Proceedings  of  HCI  International  2001,  609-613. 

Good,  M.  D.,  &  Gilkey,  R.  H.  (1996).  Sound  localization  in  noise:  I.  Effects  of  signal-to-noise  ratio. 
Journal  of  the  Acoustical  Society  of  America,  99,  1108-1117. 

Lorenzi,  C.,  Gatehouse,  S.,  &  Lever,  C.  (1999).  Sound  localization  in  noise  in  normal-hearing  listeners. 
Journal  of  the  Acoustical  Society  of  America,  105,  1810-1820. 

Simpson,  B.  D.  (2002).  The  localization  of  phantom  auditory  images  (Unpublished  mater's  thesis). 
Wright  State  University,  Dayton,  OH. 

Simpson,  B.  D.  (2011)  Sound  localization  in  multisource  environments:  The  role  of  stimulus  onset 
asynchrony  and  spatial  uncertainty  (Doctoral  dissertation).  Retrieved  from 
www.ohiolink.edu/edt/ 

Miller,  J.D.  and  Wenzel,  E.M.  (2002)  Recent  Developments  in  SLAB:  A  Software-Based  System  for 
Interactive  Spatial  Sound  Synthesis.  Proceedings  of  the  International  Conference  on  Auditory 
Display,  ICAD  2002,  Kyoto,  Japan,  pp.  403-408. 

Wenzel,  E.M.,  Miller,  J.D.,  and  Abel,  J.S.  (2000)  A  software-based  system  for  interactive  spatial 

sound  synthesis.  Proceedings  of  the  International  Conference  on  Auditory  Display,  ICAD  2000, 
Atlanta,  GA,  pp.  151-156. 

Wightman,  F.  L.  &  Kistler,  D.  J.  (1997).  Factors  affecting  the  relative  salience  of  localization  cues.  In 
R,  H,  Gilkey  &  T,  R,  Anderson  (Eds.)  Binaural  and  spatial  hearing  in  real  and  virtual 
environments  (pp.  1-23).  Mahwah,  NJ:  ErlBaum. 


49 

Distribution  A:  Approved  for  public  release;  distribution  unlimited.  88ABW  Cleared  05/29/2012;  88ABW-201 2-3075. 


5.3  Section  5.3  Aurally  Aided  Visual  Threat  Acquisition 

12  2  1  1 

Eric  Robinson  ,  Brian  Simpson  ,  Victor  Finomore  ,  Jeffery  Cowgill  ,  Valerie  L  Shalin  , 
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Many  of  today’s  military  operations  take  place  in  large  urban  environment,  which  present 
unique  challenges  due  to  limited  line  of  sight  and  increased  concealment  for  enemy 
forces.  This  experiment  evaluated  the  potential  of  a  3D  audio  display  to  aid  threat 
detection  and  localization  in  a  complex  visual  environment.  Subjects  rode  as  part  of  a 
convoy  through  a  simulated  city  where  they  encountered  snipers  surrounded  by 
distracting  personnel.  We  compared  3D  audio  cues  to  verbal  descriptions  of  the  sniper’s 
location  and  to  simple  audio  warnings  of  the  presence  of  a  sniper.  Consistent  with  past 
research,  subjects  located  the  sniper  more  quickly  in  the  3D  audio  condition  compared  to 
both  the  semantic  description  and  simple  warning  conditions.  The  3D  audio  display  was 
faster  than  the  other  displays  across  all  azimuths  but,  in  contrast  to  previous  findings,  the 
advantage  did  not  increase  with  increasing  azimuth. 
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INTRODUCTION 


Much  of  today's  military  operations  are  conducted  by  ground-based  personnel  carrying  out  missions 
in  and  around  large  urban  environments.  This  new  battlespace  poses  unique  challenges  to  operators,  as 
they  are  forced  to  fight  among  dense  populations,  where  the  distinctions  between  the  appearance  of 
enemy  fighters  and  those  who  are  friendly  and/or  neutral  are  less  clear,  and  unintended  consequences 
including  collateral  damage  and  fratricide  are  more  likely  to  occur.  Large  structures  can  obscure  the 
line-of-sight  required  to  observe  critical  activities,  and  also  provide  convenient  hiding  places  for  snipers 
and  other  enemy  forces.  In  addition,  things  change  rapidly  and  unpredictably  in  these  environments, 
and  thus  it  is  critical  for  operators  to  remain  on  high  alert  in  order  to  maintain  a  sufficient  level  of 
situation  awareness  (SA). 

In  an  environment  where  providing  timely  and  unambiguous  information  can  mean  the  difference 
between  mission  success  and  mission  failure,  it  is  critical  that  information  displays  are  not  only 
meaningful  but,  importantly,  are  attended  to  by  the  operator.  The  need  for  operators  to  remain  hands¬ 
free  and  eyes-out,  however,  suggests  that  traditional  approaches  to  information  display,  which  tend  to 
emphasize  the  presentation  of  information  through  the  visual  modality,  may  not  be  the  most 
appropriate,  for  they  require  the  operator's  visual  attention  (thus  reducing  the  time  spent  attending  to 
events  in  the  actual  environment)  and  a  transformation  may  need  to  take  place  when  going  from  a  2- 
dimensional  visual  display  to  a  3-dimensional  world.  The  use  of  advanced  auditory  display  technologies, 
which  can  provide  directional  information  indicating  the  location  of  threats  in  the  immediate 
environment,  can  be  an  effective  means  of  directing  an  operator's  attention  to  specific  locations  in 
space.  Such  displays  are  intuitive,  unambiguous,  lead  to  rapid  response  times,  and  can  be  effective  even 
when  the  operator  is  engaged  in  other  concurrent  activities  (e.g.,  visually  scanning  the  environment). 

Ephrem,  Finomore,  Gilkey,  Newton,  Romigh,  Simpson,  Brungart,  and  Cowgill  (2009)  reported  that 
subjects  responded  to  threats  more  effectively  when  using  a  3D  audio  display  to  cue  threat  location  as 
compared  to  when  using  a  monaural  display.  These  results  are  compatible  with  more  basic  studies  in 
the  spatial  hearing  literature  (e.g.,  Perrott,  Cisneros,  McKinley,  &  D'Angelo,  1996)  demonstrating 
dramatic  reductions  in  visual  search  times  when  a  colocated  auditory  cue  is  available.  However,  in  the 
experiment  of  Ephrem  et  al.,  the  3D  audio  display  also  provided  navigation  information.  Potentially,  the 
more  effective  navigation  information  allowed  for  extra  time  to  monitor  and  address  threats.  Indeed, 
Gilkey,  Simpson,  Brungart,  Cowgill,  and  Ephrem  (2007)  found  that  subjects  were  able  to  respond  more 
rapidly  and  more  effectively  to  threats  when  using  a  3D  audio  navigation  display  that  did  not  provide 
information  about  the  threats.  This  experiment  examined  the  effectiveness  of  3D  audio  more  directly  in 
a  single  task  scenario,  by  comparing  3D  audio  cuing  of  threat  (a  sniper)  locations  to  a  simple  monaural 
warning  without  spatial  information. 

Typically,  ground  soldiers  receive  spatial  information  via  verbal  description.  This  can  be  time 
consuming  and  ambiguous,  reducing  mission  effectiveness  and  potentially  leading  to  loss  of  life.  So,  in 
this  experiment  the  3D  audio  display  is  also  compared  to  a  monaural  warning  that  includes  a  semantic 
description  of  the  threat  location. 


METHOD 

To  evaluate  the  effectiveness  of  3D  audio  displays  for  cuing  threat  location,  we  developed  a 
synthetic  task  in  which  the  subject  "rides,"  as  part  of  a  convoy,  through  a  virtual  environment  depicting 
an  urban  terrain.  During  the  ride,  a  number  of  threat  situations  arise  and  the  subject  must  find  and 
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neutralize  these  threats  as  rapidly  as  possible.  Depending  on  the  experimental  condition,  one  of  three 
displays  alerts  the  subject  to  the  threat. 

Subjects 

Six  paid  subjects  (five  male,  one  female)  from  the  subject  panel  maintained  by  the  Battlespace 
Acoustics  Branch  at  Wright-Patterson  Air  Force  Base  participated  in  the  experiment.  They  ranged  in  age 
from  21  to  26  years  old,  and  had  normal  hearing  and  corrected  to  normal  vision. 

Apparatus 

The  experiments  were  conducted  in  the  VERITAS  facility;  this  facility  is  a  room-size,  5-projection- 
surface  (4  walls  and  a  floor)  virtual  environment  display  system.  High-resolution  (1200x1200  pixel) 
stereoscopic  images  are  rendered  with  Barco  Galaxy  NW-12  DLP  projectors  via  RealD  CrystalEyes  shutter 
glasses.  An  Intersense  900  tracking  system  monitors  the  position  of  the  head  and  handheld  Wand. 
Sounds  were  spatialized  using  individualized  head-related  transfer  functions  with  slab3d  application 
(v6.5.0;  Miller  and  Wenzel,  2002)  and  presented  through  Sennheiser  HMD-280-XQ.  headsets. 

Procedure 

Terrains.  Each  block  of  10  trials  took  place  within  one  of  nine  virtual  terrains,  each  measuring  500m  x 
500m.  Each  terrain  was  constructed  from  four  of  the  six  250-m  x  250-m  "tiles."  The  tiles  could  be  put 
together  so  that  any  edge  in  one  could  be  matched  up  with  the  edge  of  another,  regardless  of 
orientation.  Using  different  tiles  in  different  orientations,  a  large  number  of  varying  terrains  could  be 
created.  The  tiles  were  created  using  Presagis  Creator  Pro  software  in  Openflight.  Each  of  the  six  tiles 
was  used  an  equal  number  of  times  in  constructing  the  nine  terrains,  but  the  location  and  orientation  of 
the  tiles  was  not  controlled  systematically. 

The  terrains  contained  varying  numbers  and  types  of  structures,  ranging  from  windowless  one-story 
sheds  to  large  five-story  structures  with  many  windows  (a  screen  shot  showing  a  representative  scene  is 
provided  in  Fig.  10).  A  loop  was  identified  through  each  terrain,  along  which  the  convoy  would  travel  (an 
aerial  view  of  one  terrain  and  loop  is  shown  in  Fig.  11).  A  total  of  30  locations  on  each  loop  were 
selected  as  potential  sniper-event  sites;  these  sites  tended  to  be  surrounded  by  buildings  with  windows 
in  most  directions. 


Figure  10.  Screen  shot  of  a  representative  terrain  from  the  subject's  viewpoint.  The  front  and  side 
walls  of  VERITAS  are  shown  in  perspective. 
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Figure  11.  An  aerial  view  of  one  of  the  nine  500-m  by  500-m  terrains,  with  the  path  traversed  by  the 
convoy  during  a  block  of  trials  shown  by  the  white  line. 

Each  terrain  was  used  for  one  block  under  each  of  the  three  audio  warning  conditions.  That  is,  each 
subject  saw  each  terrain  three  times  during  the  course  of  the  experiment.  We  were  concerned  that 
subjects  would  learn  the  terrains  and  thus  be  able  to  anticipate  when  a  sniper  event  might  occur.  In 
order  to  reduce  the  impact  of  such  learning  effects,  the  subjects  experienced  the  terrains  at  three 
different  times  of  day  (dawn,  noon,  and  dusk).  These  different  times  of  day  resulted  in  different  lighting 
levels,  which  affected  shadows  and  slightly  altered  colors  in  the  environment.  In  addition,  only  10  of  the 
30  potential  sniper-event  sites  were  used  for  a  given  audio  warning  condition,  so  that  subjects 
experienced  different  actual  sniper-event  sites  under  different  audio  warning  conditions.  Finally, 
subjects  started  at  different  points  on  the  looped  path  each  time  they  experienced  a  terrain. 

Task.  During  each  block  of  10  trials,  the  subject  moved  through  one  of  the  9  terrains  as  part  of  a 
convoy  of  two  Humvees.  The  subject  was  located  on  the  rear  Humvee.  In  order  to  make  the  convoy 
move,  the  subject  had  to  orient  her/his  head  such  that  a  cursor,  linked  to  the  head  tracker,  remained 
positioned  within  a  green  target  box  on  the  lead  Humvee  (the  green  target  box  can  be  seen  on  the  top 
on  the  lead  Humvee  in  Figure  10).  This  ensured  the  subject's  head  was  in  a  relatively  constant  position 
when  they  entered  a  sniper  event  site. 

When  the  convoy  entered  an  actual  sniper-event  site  (initiating  a  trial),  the  convoy  stopped,  the 
head-linked  cursor  disappeared,  exactly  one  sniper  and  a  number  of  distracters  appeared  in  the 
windows  of  buildings  surrounding  the  convoy,  and  one  of  the  three  types  (depending  on  condition)  of 
auditory  warnings  sounded.  Typically,  the  number  of  distracters  was  50,  but  ranged  from  10  to  50 
depending  on  the  windows  available  at  any  given  site  (M  =  43.6  distracters).  The  sniper  looked  like  an 
insurgent,  and  the  distracters  looked  like  US  military  personnel  (see  Figure  12).  Subjects  were  instructed 
to  locate  and  shoot  the  sniper  (by  pointing  a  hand-held  wand  and  pulling  a  trigger)  as  rapidly  as  possible 
without  shooting  any  US  soldiers  (a  screen  shot  of  the  subject's  view  at  the  beginning  of  a 
representative  trial  is  shown  in  Fig.  13).  The  convoy  would  not  move  on  to  the  next  trial  unless  the 
sniper  had  been  shot.  When  the  sniper  was  shot,  the  audio  cue  stopped  and  the  images  in  the  windows 
disappeared.  The  head  tracked  cursor  reappeared  and  the  convoy  would  continue  along  the  path  when 
the  subject  placed  the  cursor  in  the  target  box  on  the  lead  Humvee. 
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Figure  12.  Image  of  the  target  (Left)  and  distracters  (Right). 


Figure  13.  (Top)  Screen  shot  of  the  subject's  view  at  the  beginning  of  a  trial.  The  front  and  side  walls 
of  VERITAS  are  shown  in  perspective.  For  the  reader,  a  red  arrow  has  been  added,  indicating  the 
location  of  the  sniper.  The  distracters,  dressed  as  U.S.  service  personnel,  appear  in  many  of  the  other 
windows.  The  same  scene  is  depicted,  an  instant  earlier,  in  Figure  10  (i.e.,  immediately  before  the 
trial  began).  (Bottom)  Close  up  of  the  image  in  Panel  a,  without  perspective;  the  location  of  the 
sniper  is  again  indicated  by  the  red  arrow. 

Audio  Stimuli.  Subjects  experienced  three  audio  conditions:  monaural,  semantic,  and  3D  audio.  The 
monaural  condition  did  not  utilize  any  spatial  or  descriptive  information.  The  monaural  audio  cue  served 
simply  to  alert  the  subject  that  a  trial  had  begun.  The  cue  consisted  of  a  train  of  three  50-ms  bursts  of 
white  noise,  separated  by  25-ms  of  silence,  followed  after  25  ms  by  a  recorded  warning  (a  male  voice 
saying  "Sniper")  and  then  by  500-ms  of  silence.  This  entire  sequence  was  repeated  until  the  sniper  had 
been  shot. 
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During  the  semantic  condition,  the  cue  was  a  pre-recorded  verbal  description  of  the  sniper's  location 
read  by  a  male  talker.  These  descriptions  identified  the  building  and  window  where  the  sniper  was 
located.  For  example,  the  description  for  the  trial  depicted  in  Figure  13  was: 

"Sniper.  In  the  2  story  gray  building  on  the  second  floor,  second  window  from  the  right." 

(Note,  the  target,  and  the  distracters,  could  appear  anywhere  surrounding  the  subject  in  azimuth,  but  a 
frontal  target  location  is  shown  here  for  clarity.)  When  the  building  was  not  readily  distinguished  from 
other  buildings  in  the  vicinity,  the  descriptions  followed  a  basic  heuristic  of  identifying  a  prominent 
feature  in  the  environment  visible  to  the  subject  and  then  using  that  as  a  reference  to  guide  the  subject 
to  the  sniper's  location.  For  example: 

"Sniper.  There's  a  group  of  4  buildings  with  blue  bottoms,  in  the  second  building  from  the  right,  on  the 
second  floor,  in  the  middle." 

The  duration  of  these  recorded  descriptions  ranged  from  2.0  to  7.0  s  (M=4.1  s,  SD= 0.97  s).  The 
descriptions  were  followed  by  1.5  s  of  silence  and  then  the  sequence  repeated  until  the  sniper  had  been 
shot. 

The  cue  in  the  3D  audio  condition  conveyed  information  about  both  the  sniper's  azimuth  and 
elevation  using  directional  audio  cues.  The  source  stimulus  was  identical  to  that  used  in  the  monaural 
condition,  but  was  spatialized  to  be  colocated  with  the  sniper  using  slab3D. 

Design.  This  study  used  a  repeated  measures  design  with  audio  condition  as  the  primary 
independent  variable.  The  presentation  order  of  audio  conditions  was  counterbalanced  across  subjects. 
Stimulus  terrain,  time  of  day,  and  path  entry  point  were  intentionally  confounded  with  order  (these 
nuisance  variables  are  collectively  referred  to  as  "time  of  day").  In  other  words,  each  subject 
experienced  the  same  presentation  order  of  the  combinations  of  terrain,  time  of  day,  and  path  entry 
point,  but  these  combinations  were  paired  with  different  audio  conditions  across  subjects. 

Trial  blocks.  One  subject  participated  at  a  time.  The  experimenter  read  the  subject  the  instructions 
for  the  task  and  audio  condition  and  answered  any  questions.  Once  the  subject  received  task 
instructions,  the  subject  donned  the  vest  that  contained  the  wireless  audio  receiver/transmitter  and  put 
on  the  headphones  (with  the  head  tracker  attached  to  the  headband)  and  shutter  glasses.  The  subjects 
sat  in  a  chair  in  the  center  of  the  CAVE  and  held  the  wand. 

Each  block  of  10  trials  lasted  approximately  10  minutes.  Subjects  experienced  all  nine  terrains  within 
each  audio  condition,  one  block  per  terrain.  Each  block  contained  only  one  type  of  audio  cue,  and  all 
nine  blocks  for  a  given  condition  were  completed  before  the  next  audio  condition  was  introduced. 

Subjects  received  one  training  block  of  10  trials  at  the  start  of  each  new  audio  condition.  Sessions 
lasted  one  hour,  with  a  several  minute  break  approximately  half  way  through  the  session.  When 
subjects  required  more  than  one  session  to  complete  an  audio  condition,  they  received  a  refresher  block 
of  three  trials  to  reacquaint  them  with  the  task  at  the  start  of  each  subsequent  session.  After  the 
training/refresher  block,  subjects  experienced  the  trial  blocks  for  that  audio  warning  condition. 
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RESULTS  AND  DISCUSSION 


We  replaced  outlier  trials  that  were  more  than  3  SD  from  the  mean  for  that  particular  block  and 
subject,  with  the  average  of  the  remaining  trials  for  that  block  and  subject. 

To  evaluate  the  resulting  1,620  data  points,  we  used  a  repeated  measures  ANOVA  with  3  levels  of 
audio  warning  *  3  times  of  day  *  10  trials,  and  tested  all  effects  with  treatment-specific  participant  * 
treatment  interactions.  Audio  condition  did  not  interact  with  the  nuisance  variables:  time-of-day  or 
trial.  The  ANOVA  indicated  a  significant  main  effect  for  audio  condition  (F(2,10)  =  9.07,  p  <  .0057,  3D 
Audio:  M=  6.61  s,  SD= 6.45  s,  Semantic:  M=  9.19  s,  SD=6.57 s,  Monaural:  M=  11.474  s,  5D=10.12  s  (see 
Figure  14).  This  F  ratio  remains  significant  when  evaluated  with  the  Geisser-Greenhouse  correction, 
FCriticai(l,5)  =  6.61,  a  =  .05,  and  all  paired  comparisons  are  significant  at  a  =  .05  with  a  t-test  and  error 
terms  specific  to  the  means  in  question  (monaural  vs.  semantic,  t(540)=4.40;  semantic  vs.  3D  audio, 
t(540)  =  6.50;  and  monaural  vs.  3D  audio,  t(540)  =9.41).  Time-of-day  effects  are  significant,  most  likely 
indicating  a  practice  effect  F(2,10)  =  7.60,  p  <  .0098,  M(morning)=  10.134  s,  M(noon)=  8.711  s, 
M(afternoon)  =  8.425  s. 
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Figure  14.  Time  to  acquire  and  neutralize  the  sniper  under  each  of  the  three  audio  warning 
conditions. 

These  findings  replicate  and  extend  previous  results  that  demonstrated  dramatic  reductions  in  visual 
search  times  when  spatialized  auditory  cues  are  provided  (e.g.,  Perrot  et  al.,  1996;  Simpson,  Brungart, 
Gilkey,  Cowgill,  Dallman,  Green,  Youngblood,  and  Moore,  2004).  Simpson  et  al.  found  that  their  3D 
audio  display  led  to  more  rapid  responding  than  simple  clock  angles.  Importantly,  we  also  found 
advantages  for  3D  audio  when  compared  to  warnings  that  provided  detailed  semantic  descriptions  of 
the  sniper's  location;  these  semantic  descriptions  were  like  what  ground  soldiers  might  actually 
encounter  in  operational  settings. 

Figure  15  shows  response  time  as  a  function  of  the  absolute  value  of  the  sniper  azimuth  relative  to 
the  initial  orientation  of  the  subject's  head.  As  expected  and  in  agreement  with  the  previous  literature, 
response  times  generally  increase  with  azimuth  (e.g.,  Perrott  et  al.;  Simpson  et  al.).  However,  in 
contrast  to  previous  findings,  although  responses  are  more  rapid  with  the  3D  audio  display  at  all 
azimuths,  this  advantage  does  not  increase  systematically  with  increasing  azimuth.  It  is  not  immediately 
clear  why  this  should  be  the  case.  One  explanation  maybe  the  relatively  poor  front/back  discrimination 
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observed  in  Task  1  when  the  sounds  were  presented  in  VERITAS  (however,  this  finding  also  does  not 
have  a  clear  explanation). 


6723  8ft  '**,<=>,  -5' 

Figure  15.  Mean  length  of  time  to  acquire  and  neutralize  the  sniper  as  a  function  of  the  head- 
relative  sniper  azimuth  under  each  of  the  three  audio  warning  conditions.  Positive  and  negative 
azimuths  have  been  combined. 

Overall,  these  findings  are  encouraging  and  provide  further  evidence  for  the  potential  utility  of 
auditory  displays  in  operational  settings.  That  said,  the  somewhat  counter-intuitive  results  as  a  function 
of  azimuth,  need  to  be  more  fully  explored  and  understood  in  order  to  determine  how  best  to 
implement  and  deploy  this  technology.  For  example,  perhaps  another  cue  (e.g.,  pitch,  timbre)  could  be 
used  to  supplement  front/back  cues.  Or  perhaps  semantic  and  spatial  displays  should  be  combined. 
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5.4  Section  5.4  The  Impact  of  Spatialized  Communications  on  Team  Navigation 

Andrew  Hampton1,  Valerie  L.  Shalin1,  Eric  Robinson1,  Brian  Simpson2,  Victor  Finomore2, 
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Jeffery  Cowgill  ,  Thomas  Moore  ,  Terry  Rapoch  ,  &  Robert  Gilkey 
Wright  State  University1; 

Air  Force  Research  Laboratory,  Wright-Patterson  AFB  2; 

Wright  State  Applied  Research  Coorperation3 

This  task  examined  possible  roles  for  a  3D  audio  display  in  a  team  navigation  task.  We 
considered  two  specific  capabilities:  1)  spatializing  a  communication  channel  so  that  the  receiver  hears 
the  talker's  voice  as  arising  from  the  actual  direction  of  the  talker  and  2)  spatializing  a  communication 
channel  so  that  the  receiver  hears  the  talker's  voice  as  arising  from  the  direction  of  an  object  (such  as  a 
landmark)  that  the  talker  has  "marked"  with  a  laser  rangefinder.  Several  lines  of  research  suggest  that 
the  ability  to  point  physically  or  semantically  to  specific  locations  or  destinations  in  an  environment 
(indexicality)  facilitates  the  exchange  of  verbal  directions  for  navigation.  The  person  receiving  the 
directions  can  rely  on  relatively  automatic  perceptual-motor  processes  to  aid  the  comprehension  of 
ambiguous,  and  inherently  incomplete  symbolic  representations  of  spatial  information  (A.  Clark,  1999; 
Green  &  Hummel,  2004).  When  both  subjects  in  a  conversation  can  access  a  common  referent,  the 
shared  access  establishes  common  terminology  (Brennan  &  Clark,  1996)  and  the  referents  for  pronouns 
and  other  ambiguous  terms  (Clark  &  Wilkes-Gibbs,  1986).  In  so  doing,  subjects  can  monitor  their  own 
comprehension  by  detecting  and  correcting  misapprehension  (Richardson  &  Dale,  2005).  Moreover,  the 
talker  can  simplify  the  directions  and  thereby  reduce  both  parties'  workload  (Daniel  &  Denis,  2004). 
Thus,  the  speaker  avoids  formulating  complicated  spatial  transformations  required  to  assist  reasoning 
from  the  recipient's  perspective  (Newcombe  &  Huttenlocher,  2000)  or  otherwise  risk  adding  to  the 
recipient's  cognitive  demand  (Hanna,  Tanenhaus  &  Trueswell,  2003).  So,  for  all  of  these  reasons,  the 
ability  to  identify  unambiguously  locations  in  a  large-scale  spatial  environment  should  benefit  the 
coordinated  navigation  of  spatially  disparate  teams.  The  3D  audio  display  we  consider  identifies  the 
location  of  the  talker  (capability  1)  or  identifies  the  location  of  an  important  feature  in  the  environment 
(capability  2)  and  thus  should  lead  to  both  improved  navigation  performance  and  systematic  changes  in 
language  use. 

Much  of  the  available  research  on  collaborative  spatial  reasoning  provides  a  visual  foundation  for 
shared  access  to  the  experimental  stimuli,  and  has  only  used  experimental  paradigms  that  utilize  small- 
scale  stimuli,  such  as  maps,  blueprints,  circuit  boards  or  jigsaw  puzzles  (e.g.,  Velichokovsky,  1995). 
Further,  such  research  sometimes  confounds  two  conceptually  separable  pathways  for  monitoring 
partner  comprehension:  1)  eye-contact  between  subjects  and  2)  gaze  awareness,  that  is  awareness  of 
where  one's  partner  is  looking.  Later  research  specifically  identifies  the  advantage  of  shared  visual 
access  as  gaze  awareness  (Monk  &  Gale,  2002;  Brennan,  Chen,  Dickenson,  Neider  &  Zelinsky,  2008),  and 
suggests  that  shared  gaze  reduces  the  need  for  verbal  exchange,  which  can  itself  become  detrimental  to 
performance.  One  persisting  problematic  property  of  the  typical  small-scale  spatial  reasoning  task  is  that 
it  is  representational  and  schematic,  or  more  generally,  symbolic  and  grounded  in  discrete  objects  (e.g., 
a  map).  The  separability  of  visual-motor  processing  from  symbolic,  inferential  processing  leaves  open 
the  possibility  that  established  advantages  grounded  in  reflective,  symbolic  processes  will  not 
generalize,  or  may  in  fact  have  alternative  explanations. 

Researchers  generally  examine  large-scale  spatial  reasoning  as  an  individual  task  (Lee  &  Tversky, 
2005),  or  sequential  tasks  in  which  a  separate  guide  pre-formulates  instructions  for  the  recipient  (Denis, 
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Michon  &  Tom,  2007).  This  methodological  choice  tightly  controls  the  verbal  stimuli,  and  meets  the 
current  capabilities  of  automated  navigational  aids,  but  at  the  expense  of  collaborative  dialogue  that 
could  shed  light  on  both  the  talker's  and  recipient's  cognitive  processes.  In  actual  studies  of  on-foot 
navigation,  excerpts  from  think-aloud  protocols,  deviations,  pauses,  frequency  and  duration  of  instances 
of  consulting  the  instructions  and  sketch  maps  generally  support  a  fundamental  role  for  landmarks  in 
navigation  (Daniel  &  Denis,  2004).  Landmarks  function  at  critical  decision  points,  to  prompt  a  change  in 
direction  or  to  prevent  a  potential  divergence  from  the  optimal  route.  Relative  to  street  signs, 
landmarks  are  larger  and  less  arbitrarily  named,  resulting  in  better  memory  despite  reduced  time 
reading  the  directions.  Denis  interprets  this  pattern  as  indicative  of  the  recipient's  underlying,  mental 
representation  of  symbols  standing  for  features  in  the  environment. 

Other  research  questions  the  language-based  symbolic  mediation  of  spatial  reasoning  (Newcombe  & 
Huttenlocher,  2000;  Boroditsky  &  Prinz,  2008).  For  example,  language  provides  a  fairly  coarse 
representation  of  continuously-valued  space,  and  entails  ambiguity  both  in  the  specification  and 
interpretation  of  relationships  depending  upon  the  frame  of  reference.  Such  problems  clearly  do  not 
preclude  the  use  of  language  to  support  spatial  reasoning,  but  they  do  call  into  question  a  foundational 
role  for  language  as  a  mediator  of  spatially-oriented  action.  And  so,  different  or  supplemental  means  of 
communicating  spatial  information  may  be  beneficial,  particularly  in  large  scale  spatial  tasks  in  which 
shared  visual  access  is  not  readily  available. 

We  developed  a  distributed,  shared  virtual  environment  with  large  urban  terrains  to  examine  large- 
scale  coordinated  navigation.  Two-member  teams  tried  to  find  each  other  within  these  urban 
landscapes.  The  primary  medium  of  shared  access  is  auditory  rather  than  visual.  Under  the  monaural 
condition  subjects  must  rely  on  language  to  communicate  spatial  information,  but  in  the  3D  audio 
condition  they  also  have  spatialized  communications  to  point  to  themselves  or  to  another  object  such  as 
a  landmark.  In  addition,  the  task  involves  "movement"  through  physical  space,  rather  than  abstract 
operation  on  a  schematic.  Under  some  conditions,  additional  prominent  landmarks  augment  the 
terrains,  enabling  examination  of  the  landmark  benefit  and  the  possible  interaction  with  the  audio 
condition.  That  is,  the  more  impoverished  monaural  audio  condition  could  make  subjects  more 
dependent  upon  landmarks,  while  the  experimental  3D  audio  condition  could  render  the  landmarks 
unnecessary. 

Method 

To  examine  coordinated  navigation  and  team  communication,  two-person  teams,  initially  separated, 
navigated  in  a  synthetic  large  urban  environment  in  order  to  rendezvous  as  rapidly  as  possible.  Using  a 
2X2  design  the  impact  of  monaural  vs.  3D  audio  displays  was  evaluated  in  terrains  with  additional 
prominent  visual  landmarks  and  in  comparable  terrains  without  such  landmarks. 

Subjects 

A  total  of  eight  paid  subjects  worked  in  teams  of  two,  with  two  all  male  teams,  one  all  female  team, 
and  one  male-female  team.  One  member  of  each  team  played  the  role  of  a  pararescue  jumper  (PJ)  and 
the  other  member  played  the  role  of  a  downed  pilot  to  be  rescued  (TBR).  The  subjects  ranged  in  age 
from  21  to  29  years  old.  All  subjects  were  from  the  subject  panel  maintained  by  the  Battlespace 
Acoustics  Branch  at  Wright-Patterson  Air  Force  Base  and  had  normal  hearing  and  corrected  to  normal 
vision. 
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Apparatus 

The  experiment  was  conducted  using  two  immersive  facilities.  The  Wright  State's  Virtual 
Environment,  Interactive  Technology,  And  Simulation  (VERITAS)  facility  at  Wright-Patterson  AFB  contains 
a  room-size,  5-projection-surface  (4  walls  and  a  floor)  virtual  environment  display  system.  High- 
resolution  (1200x1200  pixel)  stereoscopic  images  are  rendered  with  Barco  Galaxy  NW-12  DLP  projectors 
via  RealD  CrystalEyes  shutter  glasses.  An  Intersense  IS-900  tracking  system  monitors  the  position  of  the 
head  and  handheld  Wand.  The  second  was  the  Appenzeller  Visualization  Laboratory  (AVL)  operated  by 
the  Wright  State  Applied  Research  Corporation  in  the  Joshi  Research  Center  on  the  campus  of  Wright 
State  University  contains  a  room-size,  4  projection-surface  (3  walls  and  a  floor)  virtual  environment 
display  system  (iSpace,  Barco).  High-resolution  (1400x1050  pixel)  stereoscopic  images  are  rendered 
with  Barco  Galaxy  NH-12  DLP  projectors  via  RealD  CrystalEyes  shutter  glasses.  An  optical  tracking 
system  (ARRTTRACK)  monitors  the  position  of  the  head  and  handheld  wand.  VERITAS  and  AVL  are 
connected  via  an  Internet2  wide  area  network  (WAN)  connection.  We  used  the  Distributed  Interactive 
Simulation  (DIS)  (http://usl.sis.pitt.edu/wjj/otbsaf/IEEE1278.la-1998.pdf)  messaging  standard  to 
communicate  over  this  network  utilizing  a  DIS  software  router  to  send  local  DIS  messages  across  the 
WAN  to  the  other  location.  In  both  facilities,  sounds  were  spatialized  (with  individualized  head  related 
transfer  functions)  using  the  slab3d  (v6.5.0;  Miller  and  Wenzel,  2002)  and  presented  via  Sennheiser 
HMD-280-XQ.  headsets.  Close-talking  microphone  on  the  headsets  allowed  subjects  in  the  two  facilities 
to  talk  to  each  other.  The  DIS  radio  protocol  was  used  to  transmit  voice  communications  over  the 
network. 

To  "move"  in  VERITAS  the  subject  simply  pointed  the  wand  and  pushed  the  joystick  on  the  wand  in 
the  desired  direction  of  travel.  Because  the  display  system  in  AVL  does  not  include  a  rear  projection 
surface  it  was  not  viable  to  move  in  that  direction  (i.e.,  the  subject  would  not  be  able  to  "see  where  they 
were  going").  Therefore,  foot  pedals  were  used  in  AVL  that  allowed  subjects  to  rotate  the  virtual 
environment  around  them.  That  is,  instead  of  turning  to  face  the  back  wall  and  pointing  the 
wand/joystick  in  that  desired  direction  of  travel  as  a  subject  in  VERITAS  might  do,  the  subject  in  AVL 
could  turn  the  environment  with  the  foot  pedals  so  that  the  imagery  that  would  have  been  projected  on 
the  back  wall  was  projected  on  the  front  (or  a  side)  wall.  They  were  then  able  to  point  the  wand/joystick 
and  move  in  the  desired  direction.  Subjects  learned  this  system  quickly,  allowing  them  to  move  through 
the  entire  virtual  environment,  and  providing  a  substantially  immersive  experience.  In  both 
environments,  a  direction  indicator  appeared  in  the  center  of  the  base  of  all  walls,  displaying  primary 
and  secondary  compass  coordinates  (N,  NW,  W,  SW,  etc.)  to  let  the  subjects  know  what  direction  they 
were  facing. 

In  the  monaural  audio  display  condition,  the  DIS  radio  functioned  much  like  a  conventional  push-to- 
talk  radio  and  was  actuated  by  a  button  on  the  subject's  wand.  In  the  3D  audio  display  condition,  the 
communications  were  spatialized  so  that  they  were  heard  as  arising  from  the  environment  around  the 
listener.  Under  the  control  of  the  talker  (based  on  which  of  two  buttons  were  pressed  on  the  wand),  the 
talker's  voice  could  be  heard  as  coming  from  his/her  direction  or  from  the  direction  of  an  object  in  the 
environment  that  the  talker  marked  using  the  wand  (the  audio  annotation  capability).  To  do  this,  the 
talker  pressed  the  appropriate  button  on  the  wand  and  a  red  virtual  laser  extended  from  the  tip  of  the 
wand  to  "infinity."  "Shining"  the  laser  on  an  object  caused  the  talker's  voice  to  be  heard  as  arising  form 
the  direction  of  that  object.  So  that  it  was  clear  to  the  listener  which  pointing  capability  was  being  used, 
a  series  of  4  broadband  chirps  was  superimposed  on  the  communication  channel  immediately  after  the 
audio  annotation  capability  was  activated. 
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Procedure 

Each  trial  took  place  within  one  of  15  virtual  terrains,  each  measuring  500  m  x  500  m.  Each  terrain 
was  constructed  from  4  of  the  6  250-m  x  250-m  "tiles."  The  tiles  could  be  put  together  so  that  any  edge 
in  one  could  be  matched  up  with  the  edge  of  another,  regardless  of  orientation.  Using  different  tiles  in 
different  orientations,  a  large  number  of  varying  terrains  could  be  created.  The  tiles  were  created  using 
Presagis  Creator  Pro  software  in  the  Openflight  file.  Each  of  the  6  tiles  was  used  an  equal  number  of 
times  in  constructing  the  15  terrains.  Tile  location  and  orientation  of  the  tiles  was  not  systematically 
controlled.  Each  tile  contained  various  possible  travel  paths,  from  wide  streets  to  narrow  alleyways. 
Tiles  included  varying  numbers  and  types  of  structures,  ranging  from  windowless  one-story  sheds  to 
large  five-story  structures  with  many  windows.  Under  the  no  additional  landmark  (NALM)  condition,  no 
structures  were  taller  than  five  stories  (Representative  scenes  from  the  terrains  can  be  found  in  the 
Technical  Program  section).  Under  the  additional  landmark  (ALM)  condition,  additional  structures 
("landmarks")  were  added  to  these  same  15  terrains  (2  per  terrain;  spatially  separated  and  on  different 
tiles).  Whereas  the  tiles  had  industrial  and  residential  areas  that  were  either  "generic"  or  were  modeled 
based  on  a  Moroccan  City,  the  landmarks  reflected  architecturally  and  culturally  distinct  styles  (e.g.,  an 
Indian  sculpture,  a  modern  clock  tower,  a  conventional  American  water  tower,  etc.).  The  landmarks 
were  also  designed  to  be  taller  than  any  other  structure  in  the  terrains,  so  that  they  could  be  seen  from 
a  distance  and  potentially  be  used  as  points  of  reference  by  the  team  members  (Figs.  16  and  17  show 
images  of  two  of  the  landmarks).  The  landmarks  when  present  were  added  to  previously  vacant 
locations  in  the  terrains,  so  no  structures  were  deleted  in  creating  the  ALM  version  of  a  terrain  from  the 
NALM  version. 


Figure  16.  Two  screen  shot  of  an  archway  used  as  a  landmark  in  the  ALM  condition. 

In  each  terrain,  two  pairs  of  two  starting  locations  were  selected  by  arbitrarily  placing  one  subject 
near  one  edge  of  the  terrain  (not  necessarily  within  line-of-sight  of  the  edge)  and  then  placing  the  other 
subject  on  the  other  side  of  the  terrain  at  roughly  the  opposite  longitude  and  latitude.  The  subjects 
were  assigned  to  the  two  positions  (one  for  the  PJ  and  one  for  the  TBR);  reversing  the  assignments 
allowed  us  to  reuse  the  terrains  across  experimental  conditions. 

Task.  The  subjects  were  told  to  imagine  themselves  in  the  roles  of  a  downed  pilot  (TBR)  stranded 
somewhere  in  a  city  and  a  Pararescue  jumper  (PJ)  attempting  to  rescue  the  TBR.  Subjects  understood 
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that  the  environment  was  finite  and  if  they  began  a  simulation  near  a  border  of  the  map,  their  partner 
would  likely  start  near  the  opposite  border.  The  only  communication  available  to  the  subjects  was  the 
radio  system  controlled  by  their  wands.  Subjects  were  encouraged  to  let  one  another  know  where  they 
were  in  the  environment  and  what  they  were  doing  in  order  to  coordinate  a  rendezvous  as  quickly  as 
possible.  When  the  PJ  and  TBR  came  within  three  meters  of  one  another,  a  message  appeared  on  the 
front  screen  indicating  that  the  trial  was  complete. 


Figure  17.  Screen  shot  of  a  water  tower  used  as  a  landmark  in  the  ALM  condition. 


Each  terrain  contained  a  total  of  eight  systematically  deployed  hostiles.  Two  hostiles  appeared  near 
the  TBR's  starting  position  and  were  programmed  to  move  toward  that  position  via  a  route  that  would 
take  approximately  one  minute.  Two  additional,  stationary  hostiles  blocked  the  most  obvious  or  direct 
route  between  the  PJ  and  TBR  so  that  they  could  not  easily  walk  directly  to  one  another.  In  the 
landmark  condition,  one  stationary  hostile  was  placed  at  each  landmark  in  order  to  deter  the  strategy  of 
simply  meeting  directly  at  that  position.  The  remaining  stationary  hostiles  (two  in  the  landmark 
conditions,  four  in  the  no  landmark  conditions)  were  placed  in  locations  near  likely  travel  paths  for  one 
or  both  subjects.  The  hostiles  were  programmed  to  shoot  the  subjects  on  sight.  The  hostiles  had  an 
effective  firing  range  of  exactly  100  meters  and  when  they  fired  did  not  miss.  Subjects  were  instructed 
to  "do  your  absolute  best  to  avoid  getting  shot."  When  shot,  the  subjects  would  hear  a  gunshot  sound, 
which  was  presented  with  spatialized  audio  in  all  conditions,  allowing  the  subject  to  localize  the  threat 
aurally.  There  would  also  be  a  burst  of  blue  pixels  emanating  from  subject's  location,  similar  to  a  small 
firework.  However,  there  was  no  timeout  or  other  objective  penalty. 

The  PJ  was  armed  (with  a  "virtual  gun"  actuated  by  the  trigger  on  the  wand),  but  was  instructed  not 
to  fire  on  hostiles  unless  fired  upon  or  if  the  PJ  could  see  the  TBR  and  the  hostile  was  blocking  the  path 
to  the  TBR.  The  PJ  had  no  limit  on  the  range  of  his  or  her  weapon  except  for  the  limitations  of  screen 
resolution.  The  TBR  was  unarmed. 

Terrains  also  hosted  a  varied  number  of  civilian  entities,  but  all  had  at  least  20.  Civilians  appeared 
throughout  the  map,  approximately  half  wandering  randomly  and  half  stationary,  with  starting  positions 
tending  towards  the  center  of  the  map.  Certain  maps  had  areas  likely  to  host  gatherings  in  real-world 
settings  (a  mosque,  a  school,  and  a  park),  and  often  included  a  large  number  of  civilian  entities 
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wandering  around  randomly.  In  the  case  of  the  school,  this  was  true  for  child  entities  as  well  as  adult.  In 
a  small  number  of  the  trials,  groups  of  chicken  entities  also  appeared,  starting  in  one  general  area  but 
moving  randomly  throughout  the  terrain.  The  PJ  was  instructed  never  to  shoot  civilians. 

Design.  All  pairs  experienced  all  60  treatment  combinations  in  a  2  (audio  conditions)  *  2  (landmark 
conditions)  *  15  (trials)  repeated  measures  design.  The  order  of  the  four  combinations  of  conditions 
(Monaural/ALM,  Monaural/NALM,  3D-audio/ALM,  and  3D-audio/NALM)  was  partially  counterbalanced 
across  the  four  teams  as  follows.  Landmark  conditions  were  run  fifteen  trials  at  a  time  within  the  audio 
condition  so  that  both  landmark  conditions  would  be  experienced  before  switching  to  the  second  audio 
condition,  where  the  landmark  conditions  would  be  repeated. 

Trials.  One  team  of  two  subjects  participated  at  a  time.  The  experimenter  read  the  instructions  for 
the  task  and  audio  condition  to  the  subject  and  answered  any  questions.  Once  the  subjects  received  task 
instructions,  they  donned  the  vest  that  contained  the  wireless  audio  receiver/transmitter  and  put  on  the 
headphones  with  the  head  tracker  attached  to  the  headband  and  shutter  glasses.  The  subjects  sat  in 
chairs  and  held  the  wand  during  the  trails 

The  experiment  included  four  training  trials  for  each  of  the  audio  conditions,  two  that  contained 
landmarks  and  two  that  did  not.  These  trials  were  comparable  to  actual  trials  except  that  the 
teammates  were  placed  much  closer  together  and  generally  encountered  a  higher  concentration  of 
hostiles  so  that  they  would  experience,  and  later  recognize,  being  shot  more  easily  during  the  data 
collection  stage.  On  the  first  day  of  either  audio  condition,  teams  experienced  two  training  trials 
consecutively  and  were  given  an  additional  chance  to  ask  questions  before  data  collection  began.  On 
subsequent  days,  only  one  training  trial  was  conducted  before  data  collection  began.  The  experiment 
required  between  8  and  10  days  of  participation. 

Results  and  Discussion 

Time  to  rendezvous 

The  performance  measure  of  major  interest  was  the  time  required  to  complete  the  rendezvous  task. 
We  replaced  completion  time  outliers  that  were  more  than  3  SDs  from  the  mean  for  that  team  and 
condition  with  the  mean  for  that  team  and  condition.  We  examined  performance  with  an  ANOVA, 
including  Team  (1,  2,  3,  4),  Audio  Display  (monaural,  3D  audio),  Landmark  (NALM,  ALM),  and  Landmark  * 
Audio  Display,  and  with  Trial  as  a  continuously  valued  variable.  Due  to  the  low  number  of  teams,  and  the 
resulting  small  df  error  terms  in  the  classic  repeated  measures  decomposition,  we  used  an  aggregated 
subject*treatment  error  term  for  all  tests.  Single  df  manipulations  obviate  concern  for  sphericity.  The 
analysis  demonstrates  reduced  completion  times  under  the  3D  audio  condition  (F(l,232)  =  20.77,  p  < 
.0001,  M(3D  audio)  =  162.36,  M(monaural)  =  200.78  ,  with  some  concern  for  homogeneity  of  variance 
due  to  the  df  in  the  error  term  (Fmax(l,119)  =  1.51,  p  <  .05).  Results  for  individual  teams  are  shown  in 
Figure  18.  Neither  the  landmark  manipulation  nor  the  interaction  of  landmark  and  audio  display  were 
significant  (F(l,232)  =  .12,  p  <  .72,  and  F(l,232)  =  .69,  p  <  .41).  F(max)  tests  suggest  concern  for 
homogeneity  of  variance  in  the  landmark  main  effect  (Fmax  (1,116)  =  2.34,  p  <  .05)  and  particularly  the 
interaction  (Fmax  (3,59)  =  6.01,  p  <.05).  The  reduction  in  error  variance  relates  to  the  more  consistent 
response  times  of  3D  audio  and  the  ALM  conditions.  This  finding  made  it  reasonable  to  examine  simple 
effects  by  landmark  condition,  using  pooled  error  terms  unique  to  the  means  in  question.  However, 
both  ALM  and  NALM  conditions  suggest  an  advantages  for  3D  audio,  t(59)  =  5.19,  p<.05  and  t(59)  = 

2.16,  p  <  .05,  respectively. 
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Figure  18.  Average  time  to  complete  the  rendezvous  task  each  team  under  the  two  audio  conditions. 
The  black  bars  show  performance  averaged  across  teams. 


These  results  indicate  that  spatializing  a  communication  channel  can  provide  usable  information 
about  a  teammate's  location  in  a  team  navigation  task.  Although  this  task  was  explicitly  spatial  and 
implied  a  specific  use  for  the  information  provided  by  the  3D  audio  display  (i.e.,  walk  toward  your 
teammate's  voice),  we  expect  spatializing  communication  channels  to  promote  shared  situation 
awareness  and  team  coordination  in  a  variety  of  combat  situations.  That  is,  "knowing  where  your 
buddies  are"  is  likely  to  be  generally  useful.  On  the  other  hand,  casual  observations  indicate  that  the 
subjects  rarely  used  the  audio  annotation  capability,  and  when  it  was  used  the  subjects  did  not  appear 
to  be  marking  landmarks  to  help  orient  each  other  and  coordinate  their  rendezvous.  More  thorough 
analyses  of  the  current  data  should  help  clarify  the  subjects'  use  (and  lack  of  use)  of  audio  annotation. 
Perhaps  clearer  instructions  or  a  terrain  design  in  which  prominent  landmarks  were  visible  at  a  greater 
distance  would  have  encouraged  the  use  of  this  capability.  It  is  also  possible  that  the  availability  to  two 
spatial  capabilities  (marking  the  talker  location  and  marking  another  location)  may  have  been  confusing 
or  awkward  and  so  the  subjects  relied  on  just  one  of  them.  Separating  these  two  3D  audio  capabilities  in 
future  studies  may  help  clarify  this  result. 

The  lack  of  a  landmark  effect  may  simply  reflect  the  fact  that  the  small  number  of  additional 
landmarks  in  these  large  terrains  may  have  been  insufficient  to  influence  overall  performance. 

However,  the  absence  of  a  landmark  effect  in  the  outcome  measure  merits  investigation  for  two 
reasons.  First,  the  absence  of  an  effect  is  operationally  counterintuitive.  Highly  confusable  terrain  is  a 
known  challenge  in  military  navigation  (DuBois,  Shalin,  Levi  &  Borman  1997/1998)  and  the  presence  of 
additional  prominent  landmarks  was  expected  to  reduce  this  problem.  Second,  the  finding  is 
inconsistent  with  the  scholarly  literature  (e.g.,  Daniel  &  Denis,  2004). 

The  experiment  software  recorded  the  communications  between  the  PJ  and  the  TBR  as  they 
performed  the  team  navigation  task.  These  recordings  will  allow  for  a  more  fine-grained  (linguistic) 
analysis  that  may  reveal  a  landmark  effect,  or  may  reveal  compensatory  strategies  that  obviate  the  role 
of  landmarks  in  this  task.  We  plan  to  complete  such  linguistic  analyses,  but  these  analyses  are  beyond 
the  scope  of  the  current  paper. 

Use  of  communication  channels 

We  also  expect  that  subjects'  communications  would  be  different  in  the  3D  audio  conditions,  because 
there  would  be  less  need  for  complex  discussion  of  spatial  relations.  Again,  we  plan  to  conduct  linguistic 
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analyses  in  the  future  to  examine  these  communications  in  detail.  However,  some  preliminary  insight 
into  the  nature  of  these  communications  can  be  gleaned  by  considering  how  much  the  subjects  utilized 
the  communication  options  available  to  them. 
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Figure  19.  Average  duration  of  push-to-talk  button  presses  under  3D  audio  and  Monaural  conditions 
for  subjects  in  the  PJ  and  TBR  roles. 

Figure  19  compares  the  use  of  the  push-to-talk  communication  channel  under  the  monaural  condition 
to  the  use  of  the  push-to-talk  communication  channel  that  spatialized  the  talker's  voice  to  his/her 
location  under  the  3D  audio  condition.  As  can  be  seen,  the  average  duration  of  communications  (i.e., 
button  presses)  under  the  3D  audio  condition  was  shorter,  compatible  with  a  reduced  need  to  transmit 
spatial  information.  The  linguistic  analyses  may  reveal  that  subjects  used  fewer  words  with  3D  audio  or 
suspended  concern  for  simulating  the  recipient's  perspective,  as  compared  to  the  monaural  condition  in 
which  they  may  have  paused  longer,  used  more  words,  and  demonstrated  more  misunderstanding  of 
their  teammate  or  their  situation. 

Overall,  these  preliminary  results  provide  further  evidence  for  the  potential  utility  of  3D  audio  in  the 
battlespace  and  extend  the  findings  of  Gilkey  et  al.  (2007),  Ephrem  et  al.  (2008),  and  those  reported  in 
Task  4  from  situations  involving  teams  composed  of  an  airborne  operator  and  a  ground  soldier  to  teams 
composed  of  two  ground  soldiers.  The  planned  linguistic  analyses  should  help  to  clarify  how  subjects 
are  using  this  technology  and  why  we  did  not  find  an  impact  of  our  landmark  manipulation. 
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6.0  SECTION  6:  RESULTS,  CONCLUSIONS,  AND  RECOMMENDATIONS 


6.1  Introduction 

A  great  deal  was  accomplished  in  the  program  funded  by  agreement  FA8650-10-2-6048.  As  can  be  seen 
form  the  results,  everything  did  not  work  out  as  anticipated  regarding  the  performance  of  spatialized 
audio.  However,  the  tasks  that  were  completed  demonstrated  the  flexibility  and  value  in  using  virtual 
environments  to  create  compelling  environment  in  which  to  evaluate  technology,  even  using 
inexperienced  subjects  with  a  limited  amount  of  training. 

6.2  Results 

1.  Completed  3  of  4  Tasks  defined  in  the  initial  Statement  of  Work 

The  project  was  influenced  by  results  from  experiments  that  were  completed  immediately  prior 
to  the  kick-off  for  the  program.  As  noted  in  the  report,  these  raised  concerns  about  the  integrity 
of  the  auditory  environment  being  tested  and  the  scope  of  work  was  changed  making  it 
impossible  to  complete  Task  2  which  addressing  the  issue  of  ambient  noise. 

2.  Task  1 

a.  As  noted,  the  team  had  to  backtrack  to  deal  with  previously  inconsistent  psychophysical 
results. 

b.  Task  1.1  was  added  to  the  program  to  put  in  place  a  comprehensive  validation  and 
verification  procedure  for  the  audio  environment  and  to  develop  and  load  accurate 
HRTF's  for  all  new  subjects. 

c.  Despite  Task  1.1,  the  psychophysical  results  still  show  unacceptable  performance  in  the 
IVP  environment  for  front/back  and  elevation  for  all  subjects. 


3.  Task  4 

a.  This  was  a  very  thorough  evaluation  of  the  spatialized  audio  system  with  carefully 
controlled  deployment  of  threats  to  collect  1620  performance  measurements  in  a 
simulated  urban  environment. 

b.  The  results  consistent  with  Task  1  problems  in  terms  of  performance  correlated  to 
front/back  and  elevation  discrimination 

c.  The  results  showed  that  performance  using  spatialized  audio  was  improved  over  simple 
mono  and  semantic/descriptive  audio. 


4.  Task  3 

a.  The  implementation  of  an  all  DIS  environment  allowed  for  the  capture  of  dialog  between 
participants  and  although  there  was  not  adequate  time  or  resources  to  analyze  these 
they  provide  a  rich  database  for  future  projects. 
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b.  While  available,  subjects  made  limited  use  of  the  audio  annotation  features  of  the 
system,  possibly  because  it  requires  more  training  or  that  it  is  less  useful  that  simply 
having  the  participants  maintain  a  continuing  dialogue  and  so  "home  in"  on  each  other. 


5.  Despite  the  results  from  Task  1,  the  overall  results  showed  that  3D  audio  improved  performance 
in  both  Task  3  and  4. 

6.  Technical  results  were  included  in  Section  4.4  but  the  key  ones  were: 

a.  There  was  significant  enhancement  of  IVP  capabilities 

i.  Tiled  terrain 

ii.  Controller 

iii.  Scenario  Generator  that  could  provide  real  time  understanding  of  the  field  of  view 
for  a  subject  that  is  almost  impossible  to  determine  in  other  settings. 

iv.  ALF  simulation  showed  flexibility  of  the  IVP. 

b.  DIS  conversion  very  time  consuming  -  limited  value  given  the  results  of  Tasks  1  and  4 

c.  Data  complexity  managed  using  dual  architecture  -  just  adequate 

d.  Have  architecture  to  guide  future  development 

6.3  Conclusions  (lessons  Learned) 

1.  Developing  effective  baselines  against  which  to  compare  spatialized  audio  was  difficult  and  time 
consuming.  This  was  most  significant  in  Task  4  for  which  540  semantic  cues  had  to  be  developed, 
recorded  and  tested. 

2. 

3.  Task  1 

a.  Getting  effective  psychophysical  results  from  3D  audio  displays  is  still  a  challenge  in  all 
but  the  most  controlled  environments,  i.e.  the  ALF.  Developing  methods  that  will  deliver 
accurate  results  is  important  if  such  a  system  is  to  be  deployed  in  the  field. 

b.  This  Task  showed  that  having  the  correct  HRTF's  loaded  is  important  to  obtain  accurate 
localization  of  sounds.  The  fact  that  these  rae  not  developed  in  the  environment  in  which 
they  are  applied  may  contribute  to  the  results  observed  in  Task  ,1  which  reinforces  the 
need  for  more  research  on  the  ambient  noise  issue. 

c.  Development  of  ALF  analog  showed  flexibility  of  IVP  to  create  different  experimental 
environments. 

4.  Task  4 

a.  A  Scenario  Generator  capability  could  a  critical  differentiator  for  using  VR  because  it  is 
possible  to  establish  accurate  "ground  truth"  about  what  an  individual  or  group  of 
individuals  can  see. 

b.  Future  experiments  should  be  done  in  less  controlled  environments  than  the  "convoy" 
model  that  was  applied  in  this  Task. 

5.  Task  3 
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a.  The  design  did  not  show  the  potential  for  audio  display  to  offload  tasks  form  a  subject. 
The  next  round  should  build  on  the  present  work  to  focus  more  effectively  on  this.  For 
example,  knowing  where  a  team  member  is  as  a  secondary  task  in  working  to  a  more 
complex  primary  obejctove  ,  like  seizing  a  building. 

b.  Future  efforts  should  build  on  this  Task,  not  Tasks  1  and  4 

6.  Technical 

a.  Technical  development  and  support  requirements  were  significantly  under  estimated. 

b.  The  larger,  tile  based  terrains  worked  very  effectively 

c.  Management  and  analysis  of  data  from  trial  in  the  IVP,  including  recording  dialogues 
between  subjects  is  now  the  most  significant  challenge  going  forward. 

d.  Using  DIS  may  be  a  limit  in  terms  of  experimental  complexity. 

e.  The  skills  needed  to  develop  and  support  the  use  of  sophisticated  visualization 
environment,  like  the  IVP  is  very  limited  and  takes  time  to  develop. 

6.4  Recommendations 

1.  Bring  in  PJ's  a  Subject  Matter  Experts  to  work  in  the  environment  using  the  spatial  audio  display. 
The  results  warrant  the  investment  of  their  time  and  would  provide  essential  baselines,  feedback 
and  guidance  for  future  experiments. 

2.  Build  on  Task  3  for  next  round  of  experiments,  i.e.  stop  revisiting  Task  1  and  4  unless  for  ambient 
noise  experiments 

a.  Develop  primary  tasks  that  require  coordination  etc  and  use  3D  audio  to  manage  task 
loading 

b.  Create  experiments  that  force  the  use  of  annotation  to  understand  how  to  apply.  For 
example,  creating  input  from  a  source  outside  the  environment  like  a  UAV  operator  or 
Command  and  Control  Center,  which  must  involve  using  annotation. 

3.  Develop  experiments  to  examine  how  3D  audio  works  in  the  CSAR  mission  context.  Specifically, 
does  it  operate  through  perception/action  versus  cognition? 

4.  Expand  the  size  of  teams  complementing  real  subjects  with  avatars,  i.e.  digital  participants  with 
simpler  behaviors  to  increase  the  complexity  of  trials. 

5.  Develop  experiments  to  extend  use  of  spatialized  audio  to  Command  and  Control  environments, 
intelligence  analysis,  and  UAV  management.  This  could  be  a  powerful  adjunct  to  primarily  visual 
displays  and  monaural  voice  communications. 

6.  Technical  development 

a.  Complete  the  GUI  based  IEC 

b.  Develop  an  architecture  for  data  management  and  implement 

c.  Extend  terrain  tiles  to  include  elevation  and  building  interiors  with  managed  hierarchies 
of  objects 
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d.  Evaluate  alternatives  to  DIS  for  complex  gaming  and  training 

e.  Establish  a  program  at  WSU  to  train  developers  for  advanced  visualization  environments, 
like  the  IVP. 
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7.0  SECTION  7  APPENDICES 


1.  IDD  Messaging  Specifications 

2.  IEC  Specifications 

3.  Scenario  Builder  Extract  Sample 

4.  XML  Parameter  File  Sample 

5.  Terrain  Samples 

NOTE;  Copies  of  software  are  available  from  a  secure  source  repository. 
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IVP  IDD 
Ver.  10 
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Introduction 

This  document  is  the  Interface  Description  Document  (IDD)  for  the  Integrated  Visual  Platform  (IVP). 

The  communication  between  the  components  of  the  IVP  will  be  based  upon  the  DIS  protocol  (IEEE  1278 
series  of  standards).  This  document  describes  the  format  and  contents  of  DIS  PDUs  used  in  the  IVP  to 
communicate  between  various  applications. 

Messages 

The  IVP  will  use  the  DIS  CommentR  PDU  to  send  all  messages.  The  commentR  PDU  has  a  variable 
number  of  fixed  and  variable  length  fields,  which  are  allocated  differently  for  each  message  type  with  the 
exception  of  the  Fixed  Length  Slot  1.  Fixed  Length  Slot  1  always  holds  a  32-bit  integer  message  type  for 
the  rest  of  the  CommentR  PDU. 

The  following  enumerated  values  are  used  in  the  IVP  Messages: 


Table  2;  Message  IDs 


Value 

Message  Type 

0 

None 

1 

IVE  Change  Time  Message 

2 

IVE  Change  Terrain  Message 

3 

IVE  Change  Weather  Message 

4 

IVE  ACF  State  Request  Message 

5 

IVE  Display  Text  Message 

6 

IVE  Change  Position  Message 

7 

IVE  Show  Sphere  Message 

8 

IVE  Hide  Sphere  Message 

9 

IVE  Boresight  Message 

10 

IVE  Indicate  Speaker  Message 

11 

IVE  Stop  Indicate  Speaker  Message 

12 

IVE  State  Message 

13 

Visual  Ready  Message 

14 

Speaker  Indicated  Message 

15 

Wand  Tracker  Message 

16 

Head  tracker  Message 

17 

Hand  Tracker  Message 

18 

Experiment  Settings  Message 

19 

IVE  Start  Message 

20 

IVE  Pause/Freeze  Message 

21 

IVE  Stop  Message 

22 

IVE  Exit  Message 

23 

IAE  Load  HRTF 

24 

IAE  Load  Configuration 

25 

IAE  Create  Wave  Source  Message 

26 

IAE  Destroy  Sound  Source 

Message 

27 

IAE  HRTF  Status 

28 

IAE  Wave  Played  Message 

29 

Trial  Start  Message 
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30 

Trial  End  Message 

31 

IVE  Follow  Path  Message 

32 

IVE  Sniper  Trial  End 

33 

ITE  Load  Scenario 

34 

ITE  Reset  Scenario 

35 

IAE  Stop  All  Sounds 

36 

IDB  Create  New  Database 

37 

IDB  Save  Database 

38 

IDB  Request  Score 

39 

IEC  Participant  Score 

40 

IVE  Set  Motion  Model 

41 

Application  Kill 

42 

Application  Running 

For  the  Sender  ID  information  we  will  use  an  Exercise  ID  =  1,  Site  ID  =  1  for  VERITAS,  2  for  AVL.  So 
the  ID  will  be  1:(1  or  2): (Application  #  below) 


Table  3:  Application  IDs 


Value 

Application 

0 

None 

1 

IEC 

2 

iSpace  IVE 

3 

iSpace  IAE 

4 

CAVE  IVE 

5 

CAVE  IAE 

6 

ITE  (VR  Forces) 

7 

Operator  Station 

8 

CADWALL 

9 

Data  Logger 

Table  3:  Weather  Types 


Value 

Weather 

0 

Clear 

1 

Scattered 

2 

Overcast 

Table  4:  Motion  Model  Types 


Value 

Motion  Model 

1 

Rudder  Pedals 

2 

Tank 

3 

Point  and  Go 
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IEC  Messages 

Application  Launch 

Application  Launch  messages  are  an  internal  message  for  the  IEC.  The  application  launch  message 
contains  the  following  data  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 

Start  value 

Application 

ID 

Fixed 

2 

Long 

See 

Enum 

Application  ID  Enumeration 

Soft  Launch 

Fixed 

3 

Long 

BOOL 

Flag  for  soft  launch  on  execute 

Application  Start 

Application  Start  messages  are  issued  by  the  IEC  and  are  received  by  the  other  applications  in  the  IVP. 
All  applications  will  have  to  monitor  all  Application  Start  messages  and  respond  to  their  Application  ID. 
The  application  start  message  contains  the  following  data  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 

Start  value 

Application 

ID 

Fixed 

2 

Long 

See 

Enum 

Application  ID  Enumeration 

Application  Pause 

Application  Pause  messages  are  issued  by  the  IEC  and  are  received  by  the  other  applications  in  the  IVP. 
All  applications  will  have  to  monitor  all  Application  Pause  messages  and  respond  to  their  Application  ID. 
The  Application  Pause  message  contains  the  following  data  field: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 
Pause  value 

Application 

ID 

Fixed 

2 

Long 

See 

Enum 

Application  ID  Enumeration 

Application  Stop 


Application  Stop  messages  are  issued  by  the  IEC  and  are  received  by  the  other  applications  in  the  IVP. 
All  applications  will  have  to  monitor  all  Application  Stop  messages  and  respond  to  their  Application  ID. 
The  Application  Stop  message  contains  the  following  data  field: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 

Stop  value 

Application 

Fixed 

2 

Long 

See 

Application  ID  Enumeration 
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ID 

Enum 

Application  Terminate 

Application  terminate  messages  are  issued  by  the  IEC  and  are  received  by  the  other  applications  in  the 
IVP.  All  application  will  have  to  monitor  all  Application  Terminate  messages  and  respond  to  their 
Application  ID.  The  Application  Terminate  message  contains  the  following  data  field: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 
Terminate  value 

Component 

Location 

Fixed 

2 

Long 

See 

Enum 

Component  Location  Enumeration 

Application  Kill  Message 


The  Application  Kill  message  is  an  internal  message  for  the  IEC.  The  IEC  uses  the  message  when  it  wants 
to  force  the  termination  of  an  application.  The  Application  Kill  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 

Kill  Message  value. 

Application 

ID 

Fixed 

2 

Long 

See 

Enum 

See  application  enumeration. 

Experiment  Settings  Message 

The  Experiment  Settings  messages  are  issued  by  the  IEC  to  the  IVEs.  The  IEC  sends  the  message  at  the 
start  of  an  experiment  so  the  IVE  can  set  up  the  proper  data  collection  files.  The  Experiment  Settings 


message  contains  the  fo 

lowing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Subject  IDs 

Var 

1 

Char 

ASCII 

64  character  list  of  subject  ids,  coma 
delimited 

Team  Number 

Fixed 

3 

Long 

>=0 

Team  Number  of  subjects  to  run 

Block  Number 

Fixed 

4 

Long 

>=  0 

Block  Number  for  the  run 

Condition 

Number 

Fixed 

5 

Long 

>=0 

Condition  Number  for  the  run 

Experiment 

ID 

Var 

2 

Char 

ASCII 

64  character  Experiment  ID  for  the  run 

Subject  Ages 

Var 

3 

Char 

ASCII 

64  character  coma  delimited  list 

Subject  Sexes 

Var 

4 

Char 

ASCII 

64  character  coma  deilimted  list  0=male, 
l=female 
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IVE  Change  Terrain 

The  IVE  Change  Time  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Change  Terrain  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Change 
Terrain  Message  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Terrain  File 
Name 

Var 

1 

String 

ASCII 

64  character  file  name  without  extension 

IVE  Change  Weather 


The  IVE  Change  Weather  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only. 


The  IVE  Change  Terrain  message  contains  the  fol 


owing  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Change 
Weather  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Weather 

Fixed 

3 

Long 

See 

Enum 

Weather  Type  Enumeration 

IVE  State  Request 

The  IVE  ACF  State  Request  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only. 
The  IVE  Get  ACF  State  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  ACF  State 
Request  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

IVE  Display  Text 


The  IVE  Display  Text  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Display  Text  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Display 

Text  Message  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Display  Text 

Var 

1 

String 

ASCII 

128  character  text  to  display 
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IVE  Change  Position 


The  IVE  Display  Text  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 


IVE  Display  Text  message  contains  the  following 


ields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Change 
Position  Message  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

X  Position 

Fixed 

3 

Double 

Any 

X  Position  of  observer 

Y  Position 

Fixed 

4 

Double 

Any 

Y  Position  of  observer 

Z  Position 

Fixed 

5 

Double 

Any 

Z  Position  of  Observer 

Heading 

Fixed 

6 

Double 

Any 

Heading  of  observer 

Pitch 

Fixed 

7 

Double 

Any 

Pitch  of  observer 

Roll 

Fixed 

8 

Double 

Any 

Roll  of  observer 

IVE  Change  Time 

The  IVE  Change  Time  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Change  Time  contains  the  following  data  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Change 
Time  Message  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Time  Of  Day 

Fixed 

3 

Double 

[0,24) 

Time  of  day  in  decimal  hours 

IVE  Show  Sphere 

The  IVE  Show  Sphere  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Show  Sphere  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Show 

Sphere  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

IVE  Hide  Sphere 

The  IVE  Hide  Sphere  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The  IVE 
Hide  Sphere  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Hide 

Sphere  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 
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IVE  Boresight 

The  IVE  Boresight  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The  IVE 
Boresight  message  contains  the  following  fields:  _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Boresight 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Boresight  Flag 

Fixed 

3 

Long 

Bool 

False  =  turn  off,  True  =  turn  on 

IVE  Speaker  On 


The  IVE  Indicate  Speaker  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Indicate  Speaker  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Indicate 
Speaker  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Speaker 

Number 

Fixed 

3 

Long 

0-275 

Speaker  number  to  indicate  with  LEDs 

LED  value 

Fixed 

4 

Long 

0-4 

The  number  of  LEDs  to  turn  on 

IVE  Speaker  Off 


The  IVE  Indicate  Speaker  messages  are  issued  by  the  IEC  and  received  by  the  IVE  applications  only.  The 
IVE  Indicate  Speaker  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Stop 

Indicate  Speaker  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Speaker 

Number 

Fixed 

3 

Long 

0-275 

Speaker  number  to  stop  indicating  with 

LEDs 

Trial  Start  Message 

The  Trial  Start  message  is  issued  by  the  IEC  to  the  IVEs.  The  IEC  sends  the  message  at  the  start  of  a 
trial/event  so  the  IVE  can  setup  and  record  the  proper  data.  The  Trial  Start  message  contains  the 
following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Trial/Event 

Number 

Fixed 

3 

Long 

>=  1 

The  trial  or  event  number  being  started 

Trial/Event 

Fixed 

4 

Long 

>=0 

An  index  associated  with  different  types  of 
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Type 

events  or  trials  when  multiple  types  are  used 
in  the  same  experimental  block 

Trial  Stop  Message 

The  Trial  Start  message  is  issued  by  the  IEC  to  the  IVEs.  The  IEC  sends  the  message  at  the  start  of  a 
trial/event  so  the  IVE  can  setup  and  record  the  proper  data.  The  Trial  Start  message  contains  the 
following  fields: _ _ _ _ _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Trial/Event 

Number 

Fixed 

3 

Long 

>=  1 

The  trial  or  event  number  being  stopped,  if 
this  number  does  not  match  an  already 
started  trial,  then  it  is  ignored. 

Response/trial 

time 

Fixed 

4 

Double 

>=0 

The  time  the  trial  took  to  complete, 
calculated  by  the  IEC 

Follow  Path  Message 

The  Follow  Path  message  is  issued  by  the  IEC  to  the  IVEs.  The  IEC  sends  the  message  to  start  the  IVE 
following  a  defined  path  or  to  disable  a  currently  followed  path.  The  Follow  Path  message  contains  the 
following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Path  On/Off 

Fixed 

3 

Bool 

0  or  1 

If  false  any  paths  currently  being  followed 
will  be  stopped  and  the  IVE  will  return  to  the 
walk  motion  model,  if  true  the  IVE  will  load 
the  path  information  provided  and  start  the 
motion  model 

Path  Filename 

Var 

1 

Char 

ASCII 

The  file  name  of  the  path  file  containing  the 
waypoints,  the  “acf  ’  directory  is  the  assumed 
location,  256  byte  length 

Navigation 

Filename 

Var 

2 

Char 

ASCII 

The  file  name  of  the  navigation  file,  the  “acf’ 
directory  is  the  assumed  location,  256  byte 
length 

IVE  Set  Motion  Model  Message 

The  IVE  Set  Motion  Model  message  is  issued  by  the  IEC  to  the  IVE.  The  IEC  sends  the  message  when  it 
wants  the  IVE  to  change  the  current  motion  model.  The  IVE  will  set  the  motion  model  according  to  the 
parameters  of  the  message.  The  IVE  Set  Motion  Model  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Message  Type  enumeration  -  IVE  Set  Motion 
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Enum 

Model  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Motion  Model 

Fixed 

3 

Long 

See 

Enum 

Motion  model  to  activate. 

Ground 

Clamp 

Fixed 

4 

Long 

Flag 

Flag  to  enable  or  disable  ground  clamp.  0  = 
disable,  1  =  enable 

Intersection 

Test 

Fixed 

5 

Long 

Flag 

Flag  to  enable  or  disable  intersection  testing. 

0  =  disable,  1  =  enable. 

IAE  Load  HRTF  Message 

The  IAE  Load  HRTF  messages  are  issued  by  the  IEC.  The  IAE  Load  HRTF  message  contains  the 
following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IAE  Load 

HRTF  value 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

HRTF  File 

Var 

1 

String 

ACS  II 

64  char  file  name  of  the  HRTF  file  to  load 
without  path  or  extension 

IAE  Load  Configuration  Message 

The  IAE  Load  Configuration  messages  are  issued  by  the  IEC.  The  IAE  Load  Configuration  message 


contains  the  fol 

owing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IAE  Load 

HRTF  value 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Configuration 

File 

Var 

1 

String 

ACS  II 

64  char  file  name  of  the  Configuration  file  to 
load  without  path  or  extension 

IAE  Create  Wave  Source  Message 

The  IAE  Create  Wave  Source  messages  are  issued  by  the  IEC.  The  IEC  sends  the  message  whenever  a 
wave  sound  source  needs  to  be  configured  by  the  IAE  during  runtime.  When  received  the  IAE  will  set 
any  sources  matching  this  type  to  equal  this  wave  file,  or  create  a  new  one  if  no  sources  match.  The  IAE 
Create  Wave  Source  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IAE  Create 
Entity  value 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Kind 

Fixed 

3 

Long 

Any 

DIS  EntityKind 

Domain 

Fixed 

4 

Long 

Any 

DIS  Domain 

Country 

Fixed 

5 

Long 

Any 

DIS  Country 
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Category 

Fixed 

6 

Long 

Any 

DIS  Category 

Subcategory 

Fixed 

7 

Long 

Any 

DIS  Sub-category 

Specific 

Fixed 

8 

Long 

Any 

DIS  Specific 

Other 

Fixed 

9 

Long 

Any 

DIS  Extra 

Gain 

Fixed 

10 

Long 

>=0 

Gain  in  hundredths  of  dB  stored  as  an  int32 

MinDi  stance 

Fixed 

11 

Long 

>=0 

Minimum  distance  in  hundredths  of  meters 

MaxDistance 

Fixed 

12 

Long 

>=0 

Maximum  distance  in  hundredths  of  meters 

Mono 

Fixed 

13 

Long 

Bool 

Mono  =  1,  Spatial  =  0 

Loop 

Fixed 

14 

Long 

Bool 

Looping  =  1,  Not  Looping  =  0 

File 

Var 

1 

String 

UTF8 

256  byte  Filename,  including  path,  to  the 
wave  file 

IAE  Destroy  Sound  Source  Message 

The  IAE  Destroy  Entity  messages  are  issued  by  the  IEC.  The  IEC  sends  the  message  whenever  a  sound 
source  needs  to  be  destroyed  by  the.  The  IAE  Destroy  Entity  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IAE  Destroy 
Entity  value 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Kind 

Fixed 

3 

Long 

Any 

DIS  EntityKind 

Domain 

Fixed 

4 

Long 

Any 

DIS  Domain 

Country 

Fixed 

5 

Long 

Any 

DIS  Country 

Category 

Fixed 

6 

Long 

Any 

DIS  Category 

Subcategory 

Fixed 

7 

Long 

Any 

DIS  Sub-category 

Specific 

Fixed 

8 

Long 

Any 

DIS  Specific 

Other 

Fixed 

9 

Long 

Any 

DIS  Extra 

IAE  Stop  All  Sounds  Message 

The  IAE  Stop  All  Sounds  message  is  issued  by  the  IEC  to  the  IAE.  The  IEC  sends  the  message  when  it 
wants  the  IAE  to  stop  generating  all  sounds.  The  IAE  Stop  All  Sounds  message  contains  the  following 
fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IAE  Stop  All 
Sounds  Message  value. 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

IAE  Location  Enumeration 

ITE  Load  Scenario  Message 

The  ITE  Load  Scenario  message  is  issued  by  the  IEC  to  the  ITE.  The  IEC  sends  the  message  when  it 
wants  the  ITE  to  load  a  new  scenario.  The  ITE  Load  Scenario  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  ITE  Load 
Scenario  Message  value. 
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ITE  Location 

Fixed 

2 

Long 

See 

Enum 

VR  Forces  Location  Enumeration 

Path  Filename 

Var 

1 

Char 

ASCII 

The  file  name  of  the  file  containing  the 
scenario,  256  byte  length  max 

ITE  Reset  Scenario  Message 

The  ITE  Reset  Scenario  message  is  issued  by  the  IEC  to  the  ITE.  The  IEC  sends  the  message  when  it 
wants  the  ITE  to  reset  the  currently  loaded  scenario  to  the  beginning.  The  ITE  Reset  Scenario  message 


contains  the  fol 

owing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  ITE  Reset 
Scenario  Message  value. 

ITE  Location 

Fixed 

2 

Long 

See 

Enum 

VR  Forces  Location  Enumeration 

IDB  Create  New  Database  Message 

The  IDB  Create  New  Database  message  is  issued  by  the  IEC  to  the  IDB.  The  IEC  sends  the  message 
when  it  wants  the  IDB  to  create  a  new  database  with  the  specified  filename.  The  IDB  Create  New 
Database  message  contains  the  following  fields:  _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IDB  Create 

New  Database  Message  value. 

ITE  Location 

Fixed 

2 

Long 

See 

Enum 

VR  Forces  Location  Enumeration 

Path  Filename 

Var 

1 

Char 

ASCII 

The  file  name  of  the  new  database  file,  256 
byte  length  max 

IDB  Save  Database  Message 

The  IDB  Save  Database  message  is  issued  by  the  IEC  to  the  IDB.  The  IEC  sends  the  message  when  it 
wants  the  IDB  to  stop  recording  and  save  off  the  database  file.  The  IDB  Save  Database  message  contains 
the  following  fields:  _ _ _ _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IDB  Save 
Database  Message  value. 

IDB  Location 

Fixed 

2 

Long 

See 

Enum 

IDB  Location  Enumeration 

IDB  Request  Score  Message 

The  IDB  Request  Score  message  is  issued  by  the  IEC  to  the  IDB.  The  IEC  sends  the  message  when  it 
wants  the  IDB  to  calculate  the  participant  score  for  a  demo  run.  The  IDB  Request  Score  message 


contains  the  fol 

owing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Message  Type  enumeration  -  IDB  Request 
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Enum 

Score  Message  value. 

IDB  Location 

Fixed 

2 

Long 

See 

Enum 

IDB  Location  Enumeration 

General  Application  Messages 

Application  Running  Message 

The  Application  Running  message  is  sent  by  applications  to  the  IEC  when  they  have  fully  initialized  and 
are  ready  to  run.  The  IEC  uses  this  message  to  know  when  the  system  is  up  and  running.  The  Application 
Running  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Application 
Running  Message  value. 

IEC  ID 

Fixed 

2 

Long 

See 

Enum 

See  IEC  ID  in  the  application  enumeration. 

Application 

ID 

Fixed 

2 

Long 

See 

Enum 

See  application  enumeration.  Application  ID 
that  is  issuing  message. 

IVE  Messages 

Visual  Ready  Message 

The  Visual  Ready  messages  are  issued  by  the  IVE  and  received  by  the  IEC  application  only.  The  IVE 
sends  the  message  when  it  is  ready  to  run  after  receiving  a  Application  Start  message.  The  ACF  State 


message  contains  the  fo 

lowing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  Visual  Ready 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Speaker  Indicated  Message 


The  Speaker  Indicated  messages  are  issued  by  the  IVE  and  received  by  the  IEC  application  only.  The  IVE 
sends  the  message  when  the  participant  indicates  a  speaker  after  receiving  an  Indicate  Speaker  message. 


The  Speaker  Indicated  message  contains 


he  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Speaker 
Indicated  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Speaker 

Number 

Fixed 

3 

Long 

0-275 

Speaker  number  that  participant  indicated 
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ACF  State  Message 

The  ACF  State  messages  are  issued  by  the  IVE  and  received  by  the  IEC  application  only.  The  ACF  State 


message  contains  the  fo 

lowing  fields: 

ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  ACF  State 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Main  ACF 

File 

Var 

1 

String 

ACS  II 

64  char  file  name  of  the  main  ACF  file  in  use 
without  path  or  extension 

Local  ACF 

File 

Var 

2 

String 

ASCII 

64  char  file  name  of  the  local  ACF  file  in  use 
without  path  or  extension 

Wand  Tracker  Message 

The  Wand  Tracker  messages  are  issued  by  the  IVE.  The  IVE  sends  the  message  at  a  predetermined  rate. 
The  Wand  Tracker  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Wand  Tracker 
Message  value 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Trigger 

Fixed 

3 

Long 

Bool 

Trigger  status 

View  Rotate 
Left  Status 

Fixed 

4 

Long 

Bool 

View  Rotate  Left  Button  Status 

View  Rotate 
Right  Status 

Fixed 

5 

Long 

Bool 

View  Rotate  Right  Status 

Audio 

Fixed 

6 

Long 

Bool 

Audio  On  Button  Status 

Joystick  X 

Fixed 

7 

Double 

Any 

Joystick  X  value 

Joystick  Y 

Fixed 

8 

Double 

Any 

Joystick  Y  value 

X  Position 

Fixed 

9 

Double 

Any 

Wand  X  Position  value 

Y  Position 

Fixed 

10 

Double 

Any 

Wand  Y  Position  value 

Z  Position 

Fixed 

11 

Double 

Any 

Wand  Z  Position  value 

Heading 

Fixed 

12 

Double 

Any 

Wand  Heading  value 

Pitch 

Fixed 

13 

Double 

Any 

Wand  Pitch  value 

Roll 

Fixed 

14 

Double 

Any 

Wand  Roll  value 

Head  Tracker  Message 

The  Head  Tracker  messages  are  issued  by  the  IVE.  The  IVE  sends  the  message  at  a  predetermined  rate. 
The  Head  Tracker  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Head  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 
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X  Position 

Fixed 

3 

Double 

Any 

Head  X  Position  value 

Y  Position 

Fixed 

4 

Double 

Any 

Head  Y  Position  value 

Z  Position 

Fixed 

5 

Double 

Any 

Head  Z  Position  value 

Heading 

Fixed 

6 

Double 

Any 

Head  Heading  value 

Pitch 

Fixed 

7 

Double 

Any 

Head  Pitch  value 

Roll 

Fixed 

8 

Double 

Any 

Head  Roll  value 

Hand  Tracker  Message 

The  Head  Tracker  messages  are  issued  by  the  IVE.  The  IVE  sends  the  message  at  a  predetermined  rate. 
The  Head  Tracker  message  contains  the  following  fields: _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

X  Position 

Fixed 

3 

Double 

Any 

Hand  X  Position  value 

Y  Position 

Fixed 

4 

Double 

Any 

Hand  Y  Position  value 

Z  Position 

Fixed 

5 

Double 

Any 

Hand  Z  Position  value 

Heading 

Fixed 

6 

Double 

Any 

Hand  Heading  value 

Pitch 

Fixed 

7 

Double 

Any 

Hand  Pitch  value 

Roll 

Fixed 

8 

Double 

Any 

Hand  Roll  value 

IVE  Sniper  Trial  End  Message 

The  IVE  Sniper  Trial  End  message  is  issued  by  the  IVE  to  the  IEC.  The  IVE  sends  the  message  when  it 
detects  that  the  participant  has  hit  the  Sniper  location.  The  IVE  Sniper  Trial  End  message  contains  the 
following  fields: _ _ _ _ _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IVE  Sniper 

Trial  End  Message  value. 

IVE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Trial/Event 

Number 

Fixed 

3 

Long 

>=  1 

The  sniper  trial  number  being  ended. 

IAE  Messages 

IAE  HRTF  Status  Message 

The  IAE  HRTF  Status  Message  is  sent  out  anytime  the  IAE  loads  an  HRTF  file: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Status 

Fixed 

3 

Long 

Bool 

Success  =  1,  Failure  =  0 

HRTF 

Var 

1 

Char 

ASCII 

256  byte  HRTF  filename  and  path  of  HRTF 
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on  success.  Empty  string  on  failure. 

IAE  Wave  Played  Message 

The  IAE  Wave  Played  Message  is  sent  out  by  the  IAE  when  a  wave  sound  entity  is  first  created  to  record 
the  wave  file  used  for  that  entity: _ _ _ 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  Hand  Tracker 
Message  value. 

IAE  Location 

Fixed 

2 

Long 

See 

Enum 

Immersive  Space  Location  Enumeration 

Status 

Fixed 

3 

Long 

Bool 

Success  =  1,  Failure  =  0 

Kind 

Fixed 

4 

Long 

Any 

DIS  EntityKind  of  sound  source 

Domain 

Fixed 

5 

Long 

Any 

DIS  Domain 

Country 

Fixed 

6 

Long 

Any 

DIS  Country 

Category 

Fixed 

7 

Long 

Any 

DIS  Category 

Subcategory 

Fixed 

8 

Long 

Any 

DIS  Sub-category 

Specific 

Fixed 

9 

Long 

Any 

DIS  Specific 

Other 

Fixed 

10 

Long 

Any 

DIS  Extra 

Gain 

Fixed 

11 

Long 

>=0 

Gain  in  hundredths  of  dB  stored  as  an  int32 

MinDi  stance 

Fixed 

12 

Long 

>=0 

Minimum  distance  in  hundredths  of  meters 

MaxDistance 

Fixed 

13 

Long 

>=0 

Maximum  distance  in  hundredths  of  meters 

Mono 

Fixed 

14 

Long 

Bool 

Mono  =  1 ,  Spatial  =  0 

Loop 

Fixed 

15 

Long 

Bool 

Looping  =  1,  Not  Looping  =  0 

Wave  File 

Var 

1 

Char 

ASCII 

256  byte  Wave  filename,  including  path. 

IDB  Messages 

IEC  Participant  Score  Message 

The  IEC  Participant  message  is  issued  by  the  IDB  to  the  IEC.  The  IDB  sends  the  message  in  response  to 
an  IDB  Request  Score  message.  The  IEC  Participant  Score  message  contains  the  following  fields: 


ID 

F/V 

Slot 

Type 

Range 

Description 

Message  Type 

Fixed 

1 

Long 

See 

Enum 

Message  Type  enumeration  -  IEC  Participant 
Score  Message  value. 

IECLocation 

Fixed 

2 

Long 

See 

Enum 

IEC  Location  Enumeration 

Score 

Fixed 

3 

Long 

Pos 

Participant  Score  for  demo  run 
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1  Introduction 


1.1  Overview 

The  Integrated  Virtual  Platform  (IVP)  Experiment  Controller  (IEC)  is  the  executive  for  the  IVP  system. 

It  will  enable  the  user  to  control  the  IVP  for  an  experiment  including  configuring  and  launching  the 
appropriate  components  of  the  IVP,  maintaining  status  on  the  operation  of  the  IVP  components,  and 
shutting  down  the  IVP. 

1.2  Purpose 

This  SRS  fully  describes  the  external  behavior  of  the  IEC.  It  also  describes  nonfunctional  requirements, 
design  constraints,  and  other  factors  necessary  to  provide  a  complete  and  comprehensive  description  of 
the  requirements  for  the  IEC. 

1.3  Scope 

This  SRS  provides  a  high  level  view  of  the  operation  of  the  IEC  from  the  user  perspective.  This  should 
enable  users  get  a  view  on  how  the  software  will  control  an  IVP  experiment  and  what  kinds  of  control  and 
configuration  are  available.  Further  sections  of  the  document  will  provide  increasing  detail  on  the 
requirements  for  developers  to  use  to  create  a  design  for  the  IEC. 

1.4  Structure  and  Organization  of  This  Document 

Section  2  provides  a  high  level  view  of  the  operation  of  the  IEC.  Section  3  details  the  specific 
requirements  that  flow  from  the  high  level  operational  description. 

2  Overall  Description 

The  IEC  will  function  as  the  high  level  coordinator  of  the  systems  of  the  IVP  in  order  to  execute 
experiments.  The  IVP  is  a  system  of  systems  that  enables  multiple  participants  to  interact  inside  of  the 
same  virtual  environment.  The  IVP  is  composed  of  the  AVL  iSpace,  VERITAS  CAVE,  CADWALL,  and 
the  Operator  Station.  Both  the  iSpace  and  CAVE  are  multi-wall  immersive  virtual  environments,  while 
the  CADWALL  is  a  very  large  screen  based  display.  The  Observer  station  is  a  workstation  based  display. 
An  experiment  is  composed  of  any  or  all  of  these  systems  and  their  respective  selected  components,  the 
system  operators,  a  common  virtual  environment,  a  set  of  experiment  execution  instructions,  and  the 
experiment  data  collection  parameters. 

The  IVP  contains  the  following  systems  and  components: 

1 .  iSpace  -  immersive  system 

a.  Integrated  Virtual  Environment  (IVE)  -  visualization  of  the  virtual  environment  and  user 
input  using  Wand  or  Flystick 

b.  Integrated  Audio  Environment  (IAE)  -  3D  spatialized  audio  environment 

c.  VR-Forces  AI  -  artificial  intelligence  engine  for  all  non-participant  entities  in  the  virtual 
environment 

d.  Stealth  Viewer  -  virtual  environment  viewer 

e.  Data  Logger  -  DIS  message  logging  and  playback 

2.  CAVE  -  immersive  system 

a.  IVE  -  visualization  of  the  virtual  environment  and  user  input  using  Wand  or  Joystick 

b.  IAE  -  3D  spatialized  audio  environment 

c.  VR-Forces  AI  -  artificial  intelligence  engine  for  all  non-participant  entities  in  the  virtual 
environment 

d.  Data  Logger  -  DIS  message  logging  and  playback 
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e.  Handheld  Visual  Display  (HVD)  -  visual  compass  and  map  display  for  PJ 

3.  CADWALL  -  Very  Large  Visual  Display 

a.  Visualization 

b.  IAE  -  3D  spatialized  audio  environment 

4.  Observer  Station  -  workstation  based  display 

a.  Visualization 
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Figure  20:  IVP  System  Architecture 


The  IEC  will  have  a  communication  channel  with  each  of  the  four  main  systems.  The  four  systems 
turn  communicate  with  the  IEC  and  also  share  data  to  maintain  a  common  state  of  the  virtual 
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environment.  The  IVP  system  uses  either  DIS  or  HLA  as  the  means  by  which  the  systems  pass  messages 
to  maintain  the  virtual  environment  state.  The  IEC  communications  channel  will  be  a  virtual 
communications  channel  created  inside  the  DIS  or  HLA  data  channel  using  custom  PDUs. 

The  experiment  creator  must  choose  which  IVP  systems  to  enable  for  a  particular  experiment  and  then 
decide  which  components  of  each  system  are  appropriate.  There  are  constraints  on  which  components  are 
valid  depending  upon  which  systems  are  enabled.  For  example,  only  one  VR-Forces  or  Data  Logger  can 
be  enabled  for  an  experiment.  All  enabled  components  must  be  configured  differently  depending  on  the 
role  of  the  system  in  the  experiment.  Each  system  must  obey  scenario  control  commands  such  as 
initialize,  start,  stop,  pause  and  exit.  Further  complicating  the  configuration  problem  for  the  experimenter 
is  the  need  for  the  components  to  be  initialized  in  an  experiment  dependent  order.  This  initialization  order 
also  includes  instructions  for  the  experimenter  to  perform  actions  like  putting  the  participant  in  the  CAVE 
that  can’t  be  performed  automatically.  The  IEC  must  support  some  degree  of  variable  initialization  order 
and  instructions  to  the  experiment  so  that  multiple  experiments  can  be  supported  with  a  minimum  of 
additional  programming  effort. 
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Figure  21  :  Representative  IEC  Experiment  States 


The  Representative  IEC  Experimental  States  flow  chart  is  a  high  level  view  of  the  states  for  a  notional 
experiment  run  from  the  point  of  view  of  the  IEC.  However,  to  support  a  broad  range  of  experimental 
configurations  the  IEC  will  need  to  support  a  limited  scripting  of  experiments.  This  is  necessary  because 
each  experiment  is  different  in  its  execution  order  due  to  differences  in  equipment  configuration, 
interactions  with  the  participant,  and  configuration  of  supporting  software  applications.  The  IEC  must 
also  be  able  to  display  experimenter  prompts  to  guide  execution  of  experiments.  A  means  of  varying  the 
startup  sequence  of  the  components  in  conjunction  with  the  instruction  sequence  will  support  a  wide 
variety  of  experimental  configurations.  The  repeating  execution  loop  of  the  experiment  reflects  running  a 
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participant  through  a  block  of  trials  composed  of  different  terrain  and  threats  so  the  IEC  must  also  be  able 
to  maintain  a  list  of  the  scenarios  for  a  particular  block  of  trials. 

The  IEC  provides  a  solution  to  the  experimental  configuration,  execution  and  system  dependency 
problems  that  exist  in  the  IVP.  It  will  be  a  single  interface  to  select  systems  for  demonstrations  or 
experiments,  configure  the  components  of  the  systems  and  establish  the  execution  order  of  experiments. 
Experimental  configurations  can  easily  be  saved  and  restored.  During  an  experiment’s  execution,  the  IEC 
will  aid  the  experimenter  by  providing  execution  prompts  and  monitoring  the  status  of  the  systems.  Data 
collection  will  be  automatically  configured  so  that  all  data  for  a  trial  is  saved  in  a  coherent  manner. 

3  Specific  Requirements 

The  IEC  specific  requirements  are  divided  into  functional  requirements,  usability  requirements, 
performance  requirements,  supportability  requirements,  design  constraints,  documentation  requirements, 
purchased  components  and  interface  requirements. 

3.1  Functionality 

The  EEC’s  main  purpose  is  to  allow  the  establishment  of  experimental  configurations,  establishing  the 
order  of  initialization  of  components,  providing  sequencing  of  instructions  and  execution  of  commands 
during  experiments.  Creating,  saving,  loading  and  executing  these  different  experiments  in  an  easy  to  use 
manner  that  aids  the  experimenter  defines  the  highest  level  requirement  for  the  IEC. 

The  IEC  will  perform  four  basic  tasks  for  the  components  of  the  systems: 

1 .  Lanuch  the  component 

2.  Configure  the  component 

3.  Command  the  component 

4.  Monitor  the  status  of  the  component 

A  requirement  to  perform  each  of  the  four  tasks  will  be  listed  for  each  component.  Following  this,  any 
component  specific  aspects  of  these  four  tasks  will  be  listed  as  sub-requirements.  Because  many  of  the 
components  are  third  party  applications,  in  many  cases  it  is  not  possible  to  have  the  fine  level  of  control 
one  might  desire  over  the  components  behavior.  Regardless  of  the  level  of  support  that  a  particular 
component  might  provide  for  implementing  an  IEC  requirement,  the  requirement  will  be  listed  to  show 
what  the  desired  behavior  would  be. 

In  general,  the  IEC  will  configure  the  components  by  copy  any  necessary  configuration  files  to  the 
appropriate  directory  and  automatically  launching  the  application  if  possible. 

Commanding  components  from  the  IEC  will  be  restricted  to  Launch,  Kill,  Initialize,  Start,  Pause,  Stop 
and  Terminate  commands.  These  commands  are  sent  over  a  virtual  command  channel  that  will  utilize  the 
existing  DIS/HLA  communications  channel  to  distribute  the  commands. 

The  IEC  will  issue  specific  commands  to  components  in  order  to  execute  a  particular.  These  specific 
commands  are  listed  are  listed  as  detailed  requirements.  Additionally,  in  certain  states  the  IEC  must  wait 
for  specific  event  messages  from  components. 

3.1.1  The  IEC  shall  allow  the  user  to  create  an  experiment  sequence 

The  experiment  sequence  is  an  ordered  sequence  composed  of  instructions,  commands  and  a  scenario  list. 
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3.1. 1.1  The  IEC  shall  allow  the  user  to  add,  edit  and  delete  instruction  steps  from  an  experiment  sequence 

The  instructions  are  composed  of  a  text  file  and  optional  image  to  display  to  the  experimenter  during 
experiment  execution. 

3. 1.1. 2  The  IEC  shall  allow  the  user  to  add,  edit  and  delete  commands  from  an  experiment 

The  commands  available  to  the  user  are  defined  in  the  detailed  requirements  for  each  component.  In 
general,  each  component  should  support  at  least  launch  and  terminate.  The  IVE  supports  the  most 
specific  commands  due  to  its  unique  role  as  the  interface  between  the  participant  and  the  system. 

3.1. 1.3  The  IEC  shall  allow  the  user  to  add  edit  and  delete  scenarios  from  an  experiment 

A  scenario  entry  is  composed  of  the  terrain  file,  the  YF-Forces  scenario  file  and  the  start  position  of  the 
participant  in  the  terrain. 

3.1.2  The  IEC  shall  interface  with  the  AVL 

The  IEC  shall  be  able  to  enable  and  disable  the  use  of  the  entire  AVE  system  for  an  experiment  or 
demonstration.  This  is  controllable  during  experiment  definition  by  the  user  and  set  by  the  experiment 
configuration  file  during  an  experiment  run.  Demonstration  mode  will  also  allow  dynamic  enable/disable 
of  the  entire  AVL  system. 

3.1.2. 1  The  IEC  shall  interface  with  the  AVL  iSpace  IVE 

The  IEC  will  be  configured  with  the  main  system  ID  and  IP  address  of  the  IVE  in  the  iSpace  cluster.  The 
default  configuration  information  must  be  editable  through  the  registry.  This  information  must  be 
editable  in  the  interface  and  stored  with  the  experiment  definition  file. 

3.1.2. 1.1  The  IEC  shall  configure  the  iSpace  IVE 

The  iSpace  IVE  requires  the  file  locations  for  the  Main  and  Location  ACF  file.  These  files  provide  the 
parameters  for  the  visual  scene  rendering  and  the  input  devices.  The  iSpace  IVE  also  requires  an  initial 
state  to  be  set  and  an  initial  location  for  the  viewpoint  during  initialization  for  a  demo  or  experiment  run. 

3.1.2. 1.1.1  The  IEC  shall  set  the  IVE  initial  state 

The  IVE  initial  state  can  be  READY,  PRE  TRIAL,  or  RUNTRIAL.  By  default  the  IVE  starts  in 
PRE  TRIAL.  The  IEC  can  command  the  IVE  to  change  to  RUNTRIAL  from  PRE  TRIAL,  or  to 
PRE  TRIAL  from  RUN  TRIAL. 

3.1.2. 1.1.2  The  IEC  shall  set  the  IVE  start  location 

The  IVE  start  location  is  a  (X,  Y,  Z)  position  and  (h,  p,  r)  orientation.  The  IEC  shall  send  a  custom  PDU 
to  the  IVE  in  order  to  set  the  start  location  an  orientation. 

3.1.2. 1.1.3  The  ICE  shall  set  the  IVE  main  ACF  file 

The  IVE  main  ACF  contains  all  of  the  common  data  for  each  immersive  space.  The  IEC  shall  send  a 
custom  PDU  to  the  IVE  containing  the  file  name  of  the  main  ACF  file  to  load. 

3.1.2. 1.1.4  The  IEC  shall  set  the  IVE  location  ACF  file 

The  IVE  location  ACF  file  contains  all  of  the  location  specific  data  for  the  immersive  space  such  as  input 
devices. 
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3.1.2. 1.2  The  IEC  shall  command  the  iSpace  IVE 

The  IEC  shall  issue  the  standard  commands  of  initialize,  start,  pause,  stop  and  exit  to  control  the  iSpace 
IVE.  All  of  these  commands  are  sent  via  the  HLA/DIS  link  through  custom  PDUs. 

3.1.2. 1.2.1  The  IEC  shall  set  the  IVE  observer  position 

The  IEC  shall  command  the  IVE  to  move  the  participant  location  to  a  new  (X,Y,Z)  position  and  (h,  p,  r) 
orientation. 

3.1.2. 1.2.2  The  IEC  shall  request  the  current  IVE  configuration  ACE  file  name 

The  IEC  shall  send  a  custom  PDU  to  the  IVE  requesting  the  name  of  the  IVE’s  current  ACF  file.  The 
IEC  shall  respond  with  a  custom  PDU  containing  the  ACF  file  name. 

3.1.2. 1.2.3  The  IEC  shall  command  the  IVE  to  load  a  terrain  database 

The  IEC  shall  send  a  custom  PDU  containing  the  name  of  a  terrain  database  to  load.  This  causes  the  IVE 
to  load  the  new  terrain. 

3.1.2. 1.2.4  The  IEC  shall  command  the  IVE  to  set  the  time  of  day  of  the  environment 
The  IEC  shall  send  a  custom  PDU  to  the  IVE  to  set  the  time  of  day. 

3.1.2. 1.2.5  The  IEC  shall  command  the  IVE  to  set  the  weather  in  the  environment 

The  IEC  shall  send  a  custom  PDU  to  the  IVE  to  change  the  weather  in  the  environment.  The  valid  values 
are  clear,  scattered  or  overcast. 

3.1.2. 1.2.6  The  IEC  shall  command  the  IVE  to  display  text 

The  IEC  shall  command  the  IVE  to  display  text  on  the  front  wall  by  sending  a  custom  PDU  with  the  text 
to  be  rendered. 

3.1.2. 1.2.7  The  IEC  shall  command  the  IVE  to  exit 

The  IEC  shall  send  a  custom  PDU  to  the  IVE  commanding  it  to  exit.  The  IVE  shall  shutdown  upon 
receiving  the  command. 

3.1.2. 1.2.8  The  IEC  shall  command  the  IVE  to  start  a  trial 

The  IEC  shall  command  the  IVE  to  start  a  trial  using  a  custom  PDU.  If  the  IVE  is  in  READY  or 
PRE+TRIAL  state,  the  IVE  shall  change  its  state  to  RUNTRIAL. 

3.1.2. 1.2.9  The  IEC  shall  command  the  IVE  to  stop  a  trial 

The  IEC  will  send  command  the  IVE  to  stop  a  trial  using  a  custom  PDU.  If  the  IVE  is  in  RUN  TRIAL 
state  the  IVE  will  change  its  state  to  PRE  TRIAL.  If  the  IVE  is  not  in  RUN  TRIAL  state  no  action  is 
taken. 

3.1.2. 1.3  The  IEC  shall  monitor  the  status  of  the  AVL  iSpace  IVE 

The  IEC  will  monitor  the  status  of  the  IVE  to  the  maximum  extent  possible.  The  status  of  the  IVE 
(READY,  PRE  TRIAL,  RUN  TRIAL)  shall  be  displayed  in  the  interface  of  the  IEC.  Any  command 
acknowledgements  from  the  IVE  will  be  displayed  in  the  IEC. 
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3. 1.2.2  The  IEC  shall  interface  with  the  iSpace  IAE 

The  IEC  will  be  configured  with  the  system  ID  and  IP  address  of  the  IAE  in  the  iSpace.  This 
configuration  information  must  be  editable  through  the  registry. 

3.1.2.2.1  The  IEC  shall  configure  the  iSpace  IAE 

The  IEC  shall  provide  the  IAE  configuration  file  name  for  the  run. 

3.1.2.2.2  The  IEC  shall  command  the  iSpace  IAE 

The  IAE  needs  to  support  the  basic  commands  of  initialize,  start,  pause,  stop  and  exit. 

3.1.2.2.3  The  IEC  shall  monitor  the  status  of  the  iSpace  IAE 

Note:  there  is  no  status  available  for  the  IAE.  The  IAE  should  be  able  to  send  replies  to  configuration 
status,  runtime  status  and  command  acknowledgment. 

3.1.2.3  The  IEC  shall  interface  with  the  AVL  VR-Forces  AI 

The  VR-Forces  Remote  Control  API  is  used  to  interface  with  VR-Forces.  Note  the  only  one  VR-Forces 
AI,  either  the  iSpace  or  VERITAS,  may  be  active  for  a  particular  experiment. 

3.1.2.3.1  The  IEC  shall  configure  the  AVL  VR-Forces  AI 

The  IEC  shall  provide  the  name  of  the  configuration  file  for  the  VR-Forces  AI. 

3.1.2.3.2  The  IEC  shall  command  the  AVL  VR-Forces  AI 

The  IEC  needs  to  provide  at  least  the  initialize,  start,  stop,  pause  and  exit  commands  if  possible. 

3.1.2.3.3  The  IEC  shall  monitor  the  status  of  the  AVL  VR-Forces  AI 

TBD:  determine  what  status  is  available  from  the  VR-Forces  AI  interface. 

3.1.2.4  The  IEC  shall  interface  with  the  AVL  Stealth  Viewer 

The  IEC  should  be  able  to  remotely  launch  the  Stealth  Viewer.  There  is  no  practical  interface  with  the 
actual  application. 

3.1.2.4.1  The  IEC  shall  configure  the  AVL  Stealth  Viewer 
There  is  no  configuration  for  the  AVL  Stealth  Viewer. 

3.1.2.4.2  The  IEC  shall  command  the  AVL  Stealth  Viewer 
There  is  no  command  interface  in  the  Stealth  Viewer. 

3.1.2.5  The  IEC  shall  interface  with  the  AVL  Data  Collection  Application 

The  Data  Collection  Application  will  be  a  new  application  that  supplements  the  data  collection  performed 
by  the  MAK  Data  Logger.  Note  that  only  one  Data  Logger,  either  the  iSpace  or  the  VERITAS,  may  be 
active  for  a  particular  experiment.  The  Data  Collection  Application  shall  support  the  IEC  initialize,  start, 
pause,  stop,  and  exit  commands.  The  Data  Collection  Application  will  also  support  status  messages  to 
the  IEC. 
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3. 1.2.5. 1  The  IEC  shall  configure  the  AVL  Data  Collection  Application 
TBD:  The  configuration  interface  is  under  development. 

3.1.2.5.2  The  IEC  shall  command  the  AVL  Data  Collection  Application 
TBD:  The  command  interface  is  under  development. 

3.1.3  The  IEC  shall  interface  with  the  VERITAS 

The  IEC  shall  be  able  to  enable  and  disable  the  use  of  the  entire  VERITAS  system  for  an  experiment  or 
demonstration.  This  is  controllable  during  experiment  definition  by  the  user  and  set  by  the  experiment 
configuration  file  during  an  experiment  run.  Demonstration  mode  will  also  allow  dynamic  enable/disable 
of  the  entire  VERITAS  system. 

3.1.3. 1  The  IEC  shall  interface  with  the  VERITAS  IVE 

The  IEC  interfaces  with  the  VERITAS  IVE  in  the  same  manner  as  the  iSpace  IVE  detailed  above. 

3.1.3.2  The  IEC  shall  interface  with  the  VERITAS  IAE 

The  IEC  interfaces  with  the  VERITAS  IAE  in  the  same  manner  as  the  iSpace  IAE  detailed  above. 

3.1.3 .3  The  IEC  shall  interface  with  the  VERITAS  VR-Forces  AI 

The  IEC  interfaces  with  the  VERITAS  VR-Forces  AI  in  the  same  manner  as  the  iSpace  VR  Forces  AI 
detailed  above.  Note  the  only  one  VR-Forces  AI,  either  the  iSpace  or  VERITAS,  may  be  active  for  a 
particular  experiment. 

3. 1.3.4  The  IEC  shall  interface  with  the  VERITAS  Data  Logger 

The  IEC  interfaces  with  the  VERITAS  Data  Logger  in  the  same  manner  as  the  iSpace  Data  Logger 
detailed  above.  Note  that  only  one  Data  Logger,  either  the  iSpace  or  the  VERITAS,  may  be  active  for  a 
particular  experiment. 

3. 1.3.5  The  IEC  shall  interface  with  the  HVD 

The  HVD  is  a  handheld  computer  at  the  CAVE.  It  acts  as  a  visual  compass  and  map  display  for  the  PJ 
participant.  The  HVD  is  given  a  map  file  at  the  start  of  each  scenario  run.  Interfacing  is  performed 
through  the  DIS/HLA  network  using  custom  messages. 

3.1.3.5.1  The  IEC  shall  configure  the  HVD  terrain  display  at  the  start  of  a  run 

The  HVD  displays  an  image  of  the  terrain  with  hostile  marker,  waypoint  markers  and  an  elevation  line. 
The  HVD  listens  to  the  DIS/HLA  network  for  a  terrain  name. 

3.1.3.5.2  The  IEC  shall  configure  the  HVD  to  display  the  correct  entity  location 

The  HVD  displays  the  current  participant  location.  The  HVD  listens  to  the  DIS/HLA  network  for  the 
current  location  information. 

3.1.4  The  IEC  shall  interface  with  the  CADWALL 

TBD-  need  detail  here,  description  of  what  functions  the  CADWALL  is  supposed  to  fulfill.  Will  it  be  a 
MMD,  or  other  application? 
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3. 1.4.1  The  IEC  shall  interface  with  the  CAD  WALL  Visualization 

TBD  -  need  detail  here 

3.1.4.2  The  IEC  shall  interface  with  the  CADWALL  IAE 

TODO  -  need  detail  here 

3.1.5  The  IEC  shall  interface  with  the  Operator  Station 

The  operator  station  is  currently  the  VR-Forces  GUI,  which  monitors  the  Virtual  Environment’s  state  and 
allows  the  operator  to  insert  voice  cueing  and  control  the  experiment  run. 

3.1.5.1  The  IEC  shall  interface  with  the  Operator  Station  Visualization 

The  IEC  does  not  have  means  to  interface  with  the  VR-Forces  GUI  at  this  time. 

3.1.6  The  IEC  shall  manage  experiment  runtime  state 

There  are  three  basic  states  -  READY,  PRE  TRIAL,  and  RUNTRIAL. 

3.1.6.1  The  IEC  shall  configure  components  in  Idle  state 

The  IEC  should  be  able  to  configure  components  in  the  pre-trial  state,  but  once  the  system  is  actually 
running  a  trial  or  demo,  the  IEC  shall  have  the  configuration  disabled. 

3.1.6.2  The  IEC  shall  monitor  status  of  components  in  Running  Trial  state 

3.1.6.3  The  IEC  shall  monitor  the  status  of  components  in  Demo  state 

3. 1.6.4  The  IEC  shall  recover  from  any  errors  and  put  the  system  in  the  Idle  state 

3.1.7  The  IEC  shall  coordinate  shutdown  of  all  components 

All  components  must  eventually  be  modified  to  support  a  coordinated  clean  shutdown  procedure.  This  is 
not  a  current  capability  of  the  system  components. 

3.2  Usability 

The  IEC  has  minimum  usability  requirements  that  must  be  met.  The  purpose  of  these  requirements  is  to 
enable  a  user  with  moderate  training  to  be  able  configure  and  run  experiments  or  demonstrations  of  the 
software  within  reasonable  time  requirements. 

3.2.1  The  IEC  shall  be  usable  with  2  hours  training 

The  IEC  shall  be  usable  by  a  person  familiar  with  the  elements  of  the  system,  the  basic  requirements  to 
configure  the  system  elements  and  the  experimental  execution  parameters  with  two  hours  training. 

3.2.2  The  IEC  shall  enable  a  user  to  configure  an  existing  experiment  in  5  minutes 

A  trained  user  shall  be  able  to  modify  the  run  parameters  (block  number,  experiment  number,  trial 
number)  and  other  trial  specific  parameters  with  5  minutes  in  preparation  for  the  next  trial 

3.2.3  The  IEC  shall  enable  a  user  to  configure  a  new  experiment  in  a  day 

A  trained  user  should  be  able  to  select  the  components  for  an  experiment,  configure  the  components  and 
set  up  the  parameters  of  the  experiment  within  one  day. 
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3.2.4  The  IEC  shall  provide  the  user  with  meaningful  configuration  error  information 

The  IEC  shall  detect  configuration  errors  entered  by  the  user  and  provide  meaningful  information  on  the 
type  and  resolution  of  the  error. 

3.2.5  The  IEC  shall  conform  to  standard  Windows  interface  design  guidelines 
The  IEC  should  conform  to  all  standard  Windows  interface  guidelines. 

3.3  Performance 

The  IEC  must  be  able  to  handle  the  most  complex  configurations  of  the  IVP  and  a  bounded  number  of 
non-participant  entities  in  the  virtual  environment.  The  runtime  and  configuration  status  of  each 
component  of  the  IVP  systems  must  be  available  to  the  user  at  all  times. 

3.3.1  The  IEC  shall  display  all  system  component  errors  within  Is 

None  of  the  systems  or  their  components  has  the  ability  to  notify  the  IEC  of  errors.  In  the  future  the 
command  channel  should  be  used  to  propagate  error  messages  from  a  component  to  the  IEC. 

3.3.2  The  IEC  shall  be  able  to  run  all  IVP  systems  simultaneously 

All  available  systems  and  an  appropriate  selection  of  components  should  be  able  to  be  run 
simultaneously.  There  must  not  be  hidden  dependencies  on  what  can  and  can’t  be  run  together. 

3.3.3  The  IEC  shall  be  able  to  monitor  all  IVP  systems  simultaneously 

The  IEC  shall  monitor  all  of  the  enabled  systems  simultaneously  and  display  the  status  of  the  systems  in 
the  GUI. 

3.4  Supportability 

The  IEC  has  requirements  that  enhance  the  supportability  and  maintainability  of  the  system,  including 
coding  standards,  naming  conventions,  and  class  libraries. 

3.4.1  The  IEC  configuration  file  shall  be  in  a  custom  XML  format 

A  custom  XML  format  reflecting  the  available  configuration  items  for  the  IVP  will  be  developed.  The 
IEC  shall  use  this  custom  XML  format  to  save  configurations.  The  IEC  shall  also  read  files  using  the 
custom  XML  format  in  order  to  configure  the  IEC. 

3.4.2  The  IEC  GUI  shall  be  based  on  MFC 

The  MFC  libraries  will  provide  all  of  the  basic  GUI  functionality  for  the  IEC.  The  use  of  MFC  will 
enhance  the  supportability  of  the  IEC. 

3.4.3  The  IEC  code  shall  follow  the  project  coding  standards 

For  consistency,  the  development  team  must  agree  on  and  use  a  common  coding  standard. 

3.5  Design  Constraints 

The  IEC  has  the  following  design  constraints.  These  constraints  represent  design  decisions  that  have  been 
mandated  and  must  be  adhered  to. 

3.5.1  The  IEC  shall  be  developed  using  C++ 
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C++  will  be  the  standard  language  for  the  IEC. 

3.5.2  The  IEC  shall  be  developed  using  Microsoft  VS2008 

Microsoft  VS2008  is  being  used  for  the  IAE  and  IEC. 

3.5.3  The  IEC  GUI  shall  be  based  upon  MFC 

MFC  shall  provide  the  basic  windows  and  control  capability  required  for  the  interface  of  the  IEC. 

3.5.4  The  IEC  shall  use  the  MAK  VR-Link  library 

The  MAK  VR-Link  library  provides  the  connectivity  through  HLA  or  DIS. 

3.5.5  The  IEC  shall  use  the  MAK  VR-Forces  remote  API. 

The  MAK  VR-Forces  remote  API  allows  the  IEC  to  configure  the  VR-Forces  component. 

3.5.6  The  IEC  shall  support  communications  using  DIS 

3.5.7  The  IEC  shall  support  communications  using  HLA 

3.6  On-line  User  Documentation  and  Help  System  Requirements 

The  IEC  shall  contain  access  to  all  online  documentation  contained  in  the  WSU  IVP  wiki. 

3.7  Purchased  Components 

The  IEC  will  use  purchased  components  to  provide  necessary  functionality. 

3.7.1  The  IEC  shall  use  the  MAK  RTI 

3.7.2  The  IEC  shall  use  the  MAK  remote  API 

3.8  Interfaces 

The  IEC  supports  multiple  interfaces  with  the  user  and  various  system  and  components  of  the  IVP. 

3.8.1  User  Interfaces 

The  IEC  shall  support  multiple  user  interface  displays  to  select,  configure  and  display  the  status  of  the 
components  of  the  IVP. 

3.8. 1.1  The  IEC  shall  have  an  iSpace  configuration  user  interface 
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Experiment 

iSpace  |  CAVE  |  CAD  WALL  | 

Operator  Station  | 

Data  Logger  | 

r~  Enable  Launch  All  |  Terminate  All  | 

IVE 

I-  Enable 

Host  ID  f 

Executable  PathJ" 

Main  ACF  File  [ 

Local  ACF  File  [ 

Commands 

Launch  Terminate  |  Start  Pause  Stop 


Set  Position 

Change  Time  |  Change  T errair| 

Display  Text  | 

Get  ACF  State 

i 

Status 


VR -Forces 
I-  Enable 

Host  ID 

Executable  Path 

Configuration  File 


Commands 


Launch 

T  erminate 

Start 

Pause 

Stop 

r  Status 


3.8. 1.2  The  IEC  shall  have  a  CAVE  configuration  user  interface 
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Experiment  |  iSpace  CAVE  |  CADWALL  |  Operator  Station  |  Data  Logger  | 
Enable  Launch  All  |  T erminate  All  | 


IVE 


I-  Enable 

Host  ID  f 

Executable  Pathf 

Main  ACF  File  f 

Local  ACF  File  f 
Commands 


Launch 

Terminate  | 

Start 

Pause 

Stop 

Set  Position  |  Change  Time  |  Change  T errair]  Display  Text  | 
Get  ACF  State 


Status 


rIAE 
I-  Enable 


Host  ID 

1  ■  ■  ■  ■ 

Executable  Path 

i  ti 

Configuration  File 

i j 

Commands 


Launch  Terminate  |  Start  Pause  Stop 


Status 


rVR -Forces 
l~~  Enable 


Host  ID  f 
Executable  Path 

Configuration  File  f 
r  Commands- 


Launch 

Terminate  | 

Start 

Pause 

Stop 

r  Status- 


3.8.1.3  The  IEC  shall  have  an  experiment  configuration  user  interface 
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Experiment  |  iSpace  |  CAVE  CADWALL  |  Operator  Station  |  Datalogger] 


Experiment  ID  | 

Block  Number  |~ 
Scenario  List 


Add 

Edit 

Delete 

Experiment  Action  List 


Add  Instruction 

Add  Command 

Edit 

Delete 

“Run  Experiment 
Number  Subjects  |~ 

Start  | 


3.8.1.4  The  IEC  shall  have  a  CADWALL  configuration  user  interface 
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Experiment)  iSpace  |  CAVE  CAD  WALL  |  Operator  Station  ]  Data  Logger) 


P  Enable  Launch  All  |  Terminate  All  | 
Host  ID  |  "  "  _ 


Applications 


Add 

Edit 

Launch 

Terminate 

Delete 


□ear 


3.8.1.5  The  IEC  shall  have  a  Operator  Station  configuration  user  interface 

Experiment]  iSpace]  CAVE  CADWALL  Operator  Station  |  Data  Logger] 


Enable 

Host  Configuration 
Host  ID  f 
Executable  Path  T 


Configuration  File 


3.8.1.6  The  IEC  shall  have  a  Data  Logger  configuration  user  interface 

Experiment)  'Space  |  CAVE  |  CADWALL  |  Operator  Station  Data  Logger  | 


r  Enable 


Host  ID  f 
Executable  Path|"~ 
Configuration  File  Path|"~ 


Output  File  Pathf 
Commands 


-_i 

il8 


Launch 

Terminate 

Start 

Pause 

Stop 

3.8. 1.7  The  IEC  shall  have  a  Demonstration  user  interface 

The  demonstration  user  interface  is  contained  in  the  individual  systems  user  interfaces. 


3.8. 1.8  The  IEC  shall  have  a  IVP  status  user  interface 
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The  IVP  systems  status  is  contained  in  the  individual  systems  components  user  interface  sections. 

3.8. 1.9  The  IEC  shall  have  an  Experiment  Runtime  interface 

3.8.1.10  The  IEC  shall  have  a  Scenario  Block  Runtime  interface 

3.8.2  Software  Interfaces 

3.8.2. 1  The  IEC  shall  use  the  MAK  remote  API  to  communicate  with  MAK  VR-Forces 

3.8.2.2  The  IEC  shall  use  the  MAK  remote  API  to  communicate  with  the  MAK  Data  Logger 

3.8.3  Communications  Interfaces 

3.8.3.1  The  IEC  shall  receive  all  DIS/HLA  messages  used  by  the  system 

3.8.3.2  The  IEC  shall  communicate  with  the  system  components  using  customs  PDUs 

The  custom  PDUs  used  by  the  IEC  and  the  system  components  will  be  detailed  in  the  Command  and  Data 
ICD. 


Appendix  A:  Definitions,  Acronyms  and  Abbreviations  Used 

•  IVP  -  Integrated  Virtual  Platform  is  the  name  for  the  system  of  systems  to  allow  multiple 
participants  to  interact  in  the  same  virtual  environment. 

•  IVE  -  Integrated  Virtual  Environment  is  the  CAVE  component  of  the  iSpace  and  VERITAS 
systems. 

•  IEC  -  IVP  Experiment  Controller 

•  IAE  -  Integrated  Audio  Environment  is  the  3D  spatialized  audio  component  of  the  iSpace  and 
VERITAS  systems. 

•  HMD  -  Head  Mounted  Display  is  a  display  device  worn  on  the  head.  In  the  case  of  the  iSpace 
and  VERITAS  it  is  a  pair  of  shutter  glasses. 

•  DIS  -  Distributed  Interactive  Simulations  is  an  open  standard  for  conducting  platform  level 
simulations. 

•  HLA  -  High  Level  Architecture  is  a  general  purpose  architecture  for  distributed  simulation 
systems 

•  MFC  -  Microsoft  Foundation  Classes  is  a  class  library  used  for  creating  Windows  applications 

•  MSVS2008  -  Microsoft  Visual  Studio  2008  is  the  development  environment  for  the  software. 

•  PDU  -  Protocol  Description  Unit 

•  GUI  -  Graphical  User  Interface 

•  XML  -  extensible  Markup  Language 

•  VR-Forces  -  MAK  software  package  for  AI  control  of  non-participant  entities 

•  Data  Logger  -  MAK  software  package  for  logging  all  DIS/HLA  traffic  generated  by  the  IVP. 

•  Slab3D  -  is  a  real-time  virtual  acoustic  environment  rendering  system  originally  developed  in  the 
Spatial  Auditory  Displays  Lab  at  NASA  Ames  Research  Center. 

•  OpenFlight  -  MultiGen-Paradigm  terrain  database  file  format. 
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APPENDIX  3 


Scenario  Generation  Utility 
Requirements 


Description  - 

The  Scenario  Generator  Utility  uses  window  location  information  from  a  terrain,  the  location  of 
events,  and  mapping  information  for  azimuth  and  elevation  distribution  to  create  a  scenario  configuration 
file.  The  scenario  configuration  file  is  used  by  the  CAVE  or  iSpace  to  run  a  trial  to  run  experiments  on 
the  use  of  spatial  audio  for  threat  indication.  For  each  location,  a  threat  location  and  a  user  defined 
number  of  distractor  locations  are  chosen  based  upon  the  visibility,  distance  and  mapping  to  the  desired 
azimuths  and  elevation  for  the  experiment.  Once  all  of  the  threat  and  distractor  locations  for  a  scenario 
are  determined,  a  XML  file  that  is  readable  by  the  IEC  is  output  as  well  as  a  summary  text  file  so  the 
experiment  can  be  documented. 

Inputs  - 

1.  Window  Information  XML  File  containing  hierarchical  window  information  for  each  tile 

a.  Tile  Name  ->  Road  ->  Building  ->Windows  hierarchy  for  windows  information 

b.  Tile  Name  ->  Road  ->  Building  ->Walls  for  building  wall  information 

c.  Windows  level  will  have  position  and  orientation  for  each  window 

d.  Walls  level  will  have  the  location,  orientation  and  extents  for  each  wall 

e.  All  coordinates  will  be  in  untranslated  and  unrotated  tile  coordinates 

2.  Scenario  Description 

a.  4  Tile  Entries 

i.  Name  of  Tile 

ii.  Position  of  tile  (1-4,  mapped  to  4  quadrants  in  x-y  plane) 

iii.  Orientation  of  tile  (1-4,  mapped  to  0,  90,  180  and  270  rotations) 

b.  Number  of  Trial  Locations  -  integer  number  of  following  records 

c.  Trial  Location  Information 

i.  Tile  for  Location 

ii.  Untransformed  trial  location  (x,y) 

iii.  Untransformed  trial  orientation  (heading) 

d.  Path  Information 

e.  Event  location  information 

f.  File  generation  parameters 

3.  ALF  Speaker  Locations  File 

a.  One  line  for  each  possible  speaker  location  in  AZ  and  El 

User  Interface  - 

The  configuration  screen  details  the  tiles  available,  the  ALF  locations  configuration  file  and  4  by  4 
terrains  for  each  scenario.  For  each  source  tile,  the  user  can  define  its  ID,  the  source  file  name  and  the 
event  locations  for  the  tile.  The  event  locations  contain  the  un-translated  and  un-rotated  coordinates  of 
the  event  as  well  as  the  participants  entry  direction.  The  configuration  page  appears  below. 
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Figure  22:  Configuration  Page 

Once  the  user  has  defined  this  information  they  can  continue  to  the  processing  page.  The  processing  page 
lets  the  user  define  the  scenario  path  by  selected  the  locations  in  the  expected  order  of  execution  during  a 
trial.  In  addition,  several  other  parameters  define  the  number  of  output  files,  the  number  of  events  per 
file,  the  maximum  number  of  distractor  locations,  a  maximum  distance  and  angle  parameter,  the  default 
sound  file  and  the  tolerance  angle  for  acceptance  as  an  ALF  mapped  window.  The  processing  page 
appears  below. 
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Figure  23:  Processing  Page 

The  output  page  allows  the  user  to  specify  the  base  output  file  name  and  to  review  the  parameters  that 
have  been  entered  for  the  scenario  generation  operation.  Once  the  Generate  Scenarios  button  is  pressed 
the  utility  uses  all  of  the  information  presented  in  the  summary  to  create  the  scenario  files,  the  Output 
Page  appears  below: 
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Figure  24:  Output  Page 


Processing  - 

1 .  Program  Reads  in  the  Window  Information  file,  Scenario  Description  File  and  ALF  Speaker 


Locations  file 


2.  For  each  tile  in  the  Scenario  Description  File 

a.  Create  Transformed  Windows  information  by  applying  defined  translation  and  rotation  to 
all  entries  in  the  hierarchy  for  the  tile 

3.  For  each  Trial  Location  in  the  Scenario  Description  File 

a.  Create  Transformed  Trial  Location  Entry  by  applying  defined  translation  and  rotation  for 
appropriate  tile  to  location  and  orientation 

b.  Find  all  windows  in  LOS  and  within  maximum  distance  from  location 

i.  Clip  windows  on  distance 

ii.  Clip  windows  on  orientation 

iii.  For  each  remaining  window  check  LOS 
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1 .  Create  LOS  vector  from  trial  location  to  window 

2.  For  each  wall 

a.  Clip  walls  on  distance 

b.  Clip  walls  on  orientation 

c.  Find  intersection  of  LOS  vector  and  wall 

i.  if  intersection  is  in  bounds  of  wall,  record  wall  and  distance 
to  intersection  point 

3.  Sort  recorded  wall  intersections  by  distance 

4.  If  wall  containing  window  is  closest  then  window  is  in  LOS  and  record 
window,  otherwise  discard 

c.  Pick  sniper  location  from  windows  in  LOS 

i.  Pick  unduplicated  ALF  speaker  location  over  all  trial  locations  from  possibilities  in 
ALF  Speaker  Locations  File 

ii.  Find  closest  window  to  picked  speaker  azimuth  and  elevation 

iii.  If  no  window  within  tolerance,  try  again 

d.  Pick  n  distractor  locations  from  windows  is  LOS 

i.  Pick  unduplicated  ALF  speaker  location  from  this  trial  from  possibilities  in  ALF 
Speaker  Locations  File 

ii.  Find  closest  window  to  picked  speaker  azimuth  and  elevation 

iii.  If  no  window  within  tolerance,  try  again 


Outputs  - 

1 .  XML  file  containing  all  processing  results 

a.  Number  of  trial  locations 

b.  For  each  trial  location 

i.  All  transformed  coordinates  for  windows  in  LOS 

ii.  Transformed  trial  location  coordinates 

iii.  Transformed  sniper  window  coordinates 

iv.  All  transformed  distractor  window  coordinates 
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APPENDIX  4 


<?xml  version="1.0"  encoding=,,UTF- 8 "  ?> 

<Sniper Locations 

FileName="Z : \IVP\development\ScenarioGenerator\conf ig\Task4_lSpatial\Task4_lSpatiall . 
xml  "> 

<SniperResult  SoundFileName=" z : \IVP\development\IAE\wavs\threat_084c . wav'd 
<EventLocation> 

<SniperEventLocation  ID="Terrainl_Location3 "  TileID=f,Roundabout"  Index="2" 
LocationX="-120 . 310000"  LocationY=" 1 61 . 930000"  LocationZ=" 0 . 000000 "  OrientationX=" - 
0.707107"  OrientationY="-0 .707107"  Orient at ionZ=" 0 . 000000 "></SniperEventLocation> 
</EventLocation> 

<SniperLocation> 

<SniperEventLocation  ID="Terrainl_Location3 "  TileID="Roundabout "  lndex="0" 
LocationX="-104 . 398521 "  LocationY=" 17 9 . 495588"  LocationZ="5 . 605000"  OrientationX=" - 
1 . 000000"  OrientationY=" 0 . 000000"  OrientationZ="-0 . 000000 "></SniperEventLocation> 

</ SniperLocation> 

<SniperEventLocation  ID="Terrainl_Location3 "  TileID="Roundabout "  lndex="0" 
LocationX="-104 . 398521 "  LocationY=" 143 . 695588"  LocationZ="5 . 605000"  OrientationX=" - 
1 . 000000"  OrientationY=" 0 . 000000"  OrientationZ="-0 . 000000 "></SniperEventLocation> 
<SniperEventLocation  ID="Terrainl_Location3 "  TileID="Roundabout "  lndex="0" 
LocationX="-104 . 398521 "  LocationY=" 1 4 0 . 695588"  LocationZ="5 . 605000"  OrientationX=" - 
1.000000"  OrientationY=" 0 .000000"  OrientationZ="-0 . 000000 "></ SniperEventLocation> 
<SniperEventLocation  ID="Terrainl_Location3 "  TileID="Roundabout "  lndex="0" 
LocationX="-118 .440515"  LocationY=" 131 .867887"  LocationZ=" 1 6 . 909000" 

OrientationX=" 0 . 552097"  OrientationY=" 0 . 833780" 

OrientationZ=" 0 . 000000 "></SniperEventLocation> 

<SniperEventLocation  ID="Terrainl_Location3 "  TileID="Roundabout "  lndex="0" 
LocationX="-104 .398521"  LocationY="140 . 695588"  LocationZ="l .755000" 
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APPENDIX  5 


Terrain  2X2 

-  includes  Park,  Housing  Project,  Roundabout,  and  Industrial  (clockwise  from  Park 
bottom  left) 
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Housing  Project 
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Industrial 
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Medina 
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Park 
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Roach  2 


Road8 


Road3 


Roach 


Roundabout 
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School 
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8.0  SECTION  8:  GLOSSARY 


ALF  -  Auditory  Localization  Facility  located  at  Wright  Patterson  Air  Force  Base 
COTS  -  Commercial  Off  The  Shelf  software 
CSAR  -  Combat  Search  and  Rescue  missions 

DIS  -  Distributed  Interactive  Simulation  (DIS)  is  an  IEEE  standard  for  conducting  real-time  platform- 
level  wargaming  across  multiple  host  computers 

GPS  -  Global  Positioning  System  (GPS)  is  a  space-based  satellite  navigation  system  that  provides 
location  and  time  information 

HLA  -  A  high-level  architecture  (HLA)  is  a  general  purpose  architecture  for  distributed  computer 

simulation  systems 

IVP  -  Integrated  Virtual  Platform 

IVE  -  IVP  Visual  Environment 

IEC  -  IVP  Experiment  Controller 

IAE  -  IVP  Audio  Environment 

IDB  -  IVP  Data  Base 

IP  -  Internet  Protocol 

LED  -  Light  Emitting  Diode 

PJ  -  Pararescuemen  (AFSC  1T2X1)  also  known  as  "PJs"  (Pararescue  Jumpers)  are  United  States  Air 
Force  Special  Operations  Command  (AFSOC)  and  Air  Combat  Command  (ACC)  operatives  tasked  with 
recovery  and  medical  treatment  of  personnel  in  humanitarian  and  combat  environments 
SLAB  -  SLAB  is  a  real-time  virtual  acoustic  environment  rendering  system  originally  developed  in  the 
Spatial  Auditory  Displays  Lab  at  NASA  Ames  Research  Center. 

TCP  -  Transport  Control  Protocol 

UDP  -  User  Datagram  Protocol  (UDP)  is  one  of  the  core  members  of  the  Internet  Protocol  Suite, 

VOIP  -  Voice  Over  Internet  Protocol 

WSARC  -  Wright  State  Applied  Research  Corporation 

WSU  -  Wright  State  University 
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